Many people know that a handy data analysis feature in Excel is to create a pivot table from a spreadsheet. But spreadsheets are limited to just a million rows. You can get around this limit by jumping through a bunch of hoops.

My system lets you easily create tables with thousands of columns and hundreds of millions of rows. (Just drop a CSV, Json, or other file on a window to create a table.)

Now you can create a pivot table from it with just a few clicks of the mouse. It is fast (I created a pivot table against an 8.5 million row table of Chicago crime data in less than a second.)

The resulting pivot table is interactive. Each cell (row/column intersection) has all the row keys mapped to it. Double-click on any cell and it will instantly show you all the rows in the original table that were used to calculate the cell. You can then analyze those rows further.

It also works well against much larger tables. I have tested it out against 25M, 50M, 100M, and 200M+ row tables.

Not trying to discourage you, I am curious as to see how you are planning to enter the market as that was something I couldn’t answer when considering working on spreadsheet tools of various kinds or even an excel alternative.

But if your dataset has millions of rows and you need something quick to help you slice and dice the data in a variety of ways to try and find valuable insights in it to drive business decisions; then maybe you are looking for something better.

BTW: creating pivot tables is just one of dozens of things my system can do. I am currently trying to figure out which features will attract the most customers.

A pardon is only a protection against a 'vengeful administration' if that administration is not your party.

Pardons are only a miscarriage of justice if those pardoned don't share your ideology.

If that means a ton of literal insurrectionists go free, that's fine with me. We elected someone precisely to do that. It's on the voters if we elected someone who was literally treasonous himself.

I hope the insurrectionists take the opportunity to get on with their lives. I gather that quite a few have already been banned for other crimes, and that's too bad.

I don't want prison to be vengeance. I want prison to make us all safer. I'd like the President to take a lot of leeway in finding people who are going to be productive citizens if they were given that gift.

If there are other methods short of prison that can render law-breakers harmless - such as restrictions on certain activities and occupations - then those should be pursued first.

The ghost of this philosophy, however attenuated, can be seen in systems of pardon and parole.

I acknowledge that a desire for retribution - to punish the evil-doer; make them suffer for what they've done - is a strong impulse (I feel it myself!), deeply imbedded in our tribal psyches, but it should be fought, not indulged.

This seems to me to be the only moral basis for a system of justice and incarceration, though I have no idea how to nudge a society towards this model. Some northern European countries approach it.

I have a somewhat distant relative who was pardoned after being over-prosecuted by a zealous DA. They were a victim of a felony who did something in response that could have been charged as anything from a citation/violation to a felony, the DA’s discretion was to choose the harshest possible resolution.

They still have a hard time getting work because the conviction must be reported.

Not everyone making a political argument is engaging in cynical tribalism. Believe it or not, some people do actually believe in things.

I support them at the national level because they're the least evil of the two and exactly two relevant options available, and the one which at least gives lip service to progressive values. But that is still like supporting Mussolini over Hitler. Locally I vote third party when I have a chance.

And I live in Texas so none of my votes matter anyway.

If I claimed that I had no choice but to become a Republican, I would be justifiably laughed at (even by fellow Republicans). Political views and affiliations are certainly choices.

Anyone can claim that their opinion is the only sane one.

I explained my choice. Choosing the lesser evil is a choice. I don't think anyone in this thread besides you is getting hung up on this, and I don't know why you're being so aggressively pedantic. It's weird.

I certainly don't.

What big differences do you see?

Like most things in MAGAland, these matters are framed in a certain way, and all nuance is eliminated. The irony of being upset about Biden while being a cheerleader for Don Jr is lost.

I was pretty young and not really caring, but I recall them being unhappy about Clinton’s last minute pardons as they were obviously compromised. I recall conversations about the Gerald Ford pardon of Nixon that happened around the context of Scooter Libby and there was an acceptance that that was more of the grist of politics than anything.

I don’t recall any president in my memory proposing pardons in advance to people blatantly breaking the law. When I was younger, republicans were almost solemnly committed to the “rule of law”. That changed in the last 20 years. This president is very tellingly an admirer of Andrew Jackson, who was in most measurement a disaster, as is this admin.

Personally, I live my life fairly “conservatively”, but have more progressive politics with some “exceptions” for the modern sense of the word. I respect those who disagree with my point of view, but am not tolerant of disdain for the law, basic fairness and society.

But if the idea isn't fleshed out enough where you think that you could go through all that trouble; then maybe you are jumping the gun.

Let's call this cycle A and cycle B.

If A is too hard, miners drop out, cycle B gets easier, miners flood back, cycle A gets harder.

This results in the hard cycle getting longer and the easy cycle getting shorter.

This isn't completely critical as there is I believe a small damping effect, so it isn't completely lethal to bitcoin, but a key thing about bitcoin mining is that whether other people are mining or not doesn't actually affect your own profitiability.

Other people dropping out doesn't actually mean you get more bitcoins per hour/watt, it only affects the next difficulty adjustment as a secondary effect.

I guess you could share the power supply and cooling infra, but I am dubious the savings are enough to have half your silicon idle all the time.

Mining ASICs (Bitcoin, etc.) cannot be repurposed…they’re hardwired for a single hash algorithm and lack matrix math needed for neural networks.

Your mining rate is simply your hash rate vs the hash difficulty.

Conceptually, it's analoglous to rolling random numbers in (0,1) until you get to a number smaller than 1/X, where X is large.

How long it takes you to do that, isn't dependent on how many other people are also trying to do that, if you get 1 hit per hour, then lots of other people getting hits doesn't actually stop you getting your 1 hit per hour.

Now, that's not quite the whole truth, as there's a small amount of time needed for propagation of the previous chain, but with an average hit globally of ~10 minutes, that's not actually a big factor.

What could happen to incentivise people is increased fees if blocks get less common due to dropped miners, there'd be more competition to get into blocks if they start filling up.

That combined with the fixed costs such as depreciation as othes mentioned, keeps the risk of this form of failure to a minimum.

I have been paying for my VPN with lightning payments; it takes less than one second to go through.

One is a quick summary of the current balance in a channel. A new transaction is created each time the balance in the channel changes. It's somewhat cheap to put on the blockchain (And the main saving is that you only need to post the final update when you close the channel), but venerable to one side putting an old stale transaction onto the blockchain to profit.

The other transaction forms a chain of proof for current state, invalidating previous balance update transactions. It's somewhat expensive to post, as it will pull in the whole history.

Both peers need to continually watch the chain (or contract a 3rd party to watch) to make sure the other peer isn't cheating by posted a stale balance transaction. These special transactions are time locked, so once one is posted, you have like 24 hours to post the proof transaction and reverse it.

> What's the point of it all in this case?

Lightning is an L2 protocol, highly scalable and used for low cost payment in Bitcoin. Level 1 networks are almost never used for user transactions: your credit card payments do not go over fedwire, etc. Bitcoin protocol is not scalable to serve worldwide money transfer needs; Lightning is. And with the cost of a penny per transaction or so.

> Why not just use regular currency?

There are a lot of frictions in the current banking systems, because money laundering, because drugs, because whatever. Getting $5-$10k in regular currency while on an overseas trip can be a major quest. With Lightning I can transfer that much (or more) in a few mouse clicks.

As a side note, I think the federales are already way too nosy regarding my use of my own money, so I want to give alternative options as much business as I can. My 2c.

Fedwire isn't a "level 1 network", it's an entirely different service with different end users and goals in mind. ACH isn't an "L2 protocol", but does orders of magnitude more transactions per second than Bitcoin.

It's like cryptobros don't understand the basics of the systems they're attempting to replace.

Most people who use Lightning do not operate their own nodes; same as with other payment methods -- credit card users do not operate their own payment networks, people writing checks do not operate their own banks, etc.

It feels like we are talking across each other and I just do not get it.

1. can't allow an single village to operate purely on them, because they're too slow

2. or they're not decentralized, and the entire "democratic" angle dies with that

And then what's the point of cryptocurrencies for most people? Why not just use the "tradfi" and "fiat currencies" and use the money propping up cryptocurrencies to actually make the world a better place?

If we shut down all current cryptocurrencies and diverted resources used to actual productive uses, the world would probably end up with a net gain.

I'm just ranting. I would want cryptocurrencies to be amazing but right now they seem useful for people with cyberpunks fetishes, for criminals, and rarely, for actual regular people from fragile states (not rich people trying to exfiltrate wealth).

The point is that it is resistant to censorship, it is pseudonymous, and so on (all the other bitcoin attributes apply)

> Or that to scale everyone would need to store a petabyte size blockchain That is addressed in the whitepaper (SPVs and pruning)

Mining Bitcoin requires both hardware and electricity, and the cheapest electricity is solar. There isn't any severe scarcity of the raw materials to make solar panels, or of sunlight, so Bitcoin miners can buy as many solar panels as they want and it would only increase the economies of scale for producing them for other purposes too.

Solar has inconsistent output. There is none at night and it varies based on weather during the day. Mining hardware wants a fixed constant amount of power. The logical thing for miners to do is to somewhat overbuild the amount of generation they need and then sell any surplus to the grid, and sell to the grid during the day and buy it back at night. The same incentives hold if the miners and the generators are two different parties, and the result is to increase the amount of generation capacity by more than the amount of consumption and have "too cheap to meter" during periods of above-average generation. (You were never going to get "too cheap to meter" during periods when generation is low and demand is high.) And even during short periods when demand significantly outstrips supply, then their incentive is to stop operating those few days out of the year because the spot price of electricity makes mining unprofitable then, which allows the generation capacity installed to do mining be used to support the rest of the grid and inhibits the price of electricity from rising above the point where mining becomes unprofitable even for people who already have mining hardware. It's basically a buffer that buys electricity when it's cheap and sells when it's expensive.

Bitcoin has a volatile price. When the price is high, miners buy hardware and increase or pay someone else to increase generation capacity. When the price declines, the mining hardware becomes idle but the power generation capacity still generates fungible electricity that can be used for any other purpose. The result is that miners pay to install a lot of generation capacity during the boom, and have the incentive to prioritize investing in more generation rather than newer/more efficient mining hardware because it's the thing that's still worth something if the price declines, and that generation capacity then gets offloaded into the grid during the bust, with the result that grid prices go up some during the boom and down by even more during the bust. By the next boom some of the generation added last time has already been sold to non-miners or locked into long-term contracts so now they're back to adding new capacity again.

"Incentive to fund increases in generation capacity but then not use all of it" has what effect on average prices?

Most significantly that the increased demand due to mining will result in grid operators investing in proportional new capacity to offset it over a reasonable time scale. Instead of just driving up prices due to basic supply/demand.

Also that miners are only consuming electricity when renewables dominate the mix. Otherwise they're responsible for more CO2 emissions to do something useless.

Plus in markets like Texas, miners also manage to get subsidies intended for actually useful customers like factories to go offline at peak times. So ratepayers are essentially paying protection money so they won't over stress the grid by performing their useless work.

In a world where bitcoin miners had to install new solar capacity to entirely offset their peak usage and sell back to the grid any excess then sure, seems like that wouldn't be a big societal net negative like it is right now.

If anyone can buy a piece of land, plop down some solar panels and start selling power to the grid, that's what happens whenever the market price gets higher than the cost of doing it. If they can't, that seems more like a regulatory problem than a Bitcoin problem.

> Also that miners are only consuming electricity when renewables dominate the mix. Otherwise they're responsible for more CO2 emissions to do something useless.

~100% of new net generation capacity in the US is renewables and that seems poised to continue for economic reasons. Adding some new nuclear could also make sense in an amenable political environment (some new data centers are trying to build it) but that also doesn't emit CO2.

> Plus in markets like Texas, miners also manage to get subsidies intended for actually useful customers like factories to go offline at peak times. So ratepayers are essentially paying protection money so they won't over stress the grid by performing their useless work.

Those "subsidies" (really discounts) exist because the cost of supplying power 100% of the time is dramatically higher than the cost of supplying it 99% of the time, so you pay less if you only need it 99% of the time and the power company gets to choose when.

Having more customers that can do that allows the grid to supply power to everyone for less money. You install some solar panels whose average generation can support a Bitcoin mining operation + X number of homes. When the output is half of normal for an extended period of time, the Bitcoin mining operation cuts out and "half of normal output from twice as many panels" can power all of the homes. Without the mining operation buying half the typical output there would only be half as many panels to begin with and then you would need something like natural gas peaker plants to power the other half of the homes when solar generation is low, which both costs more and emits more CO2.

> In a world where bitcoin miners had to install new solar capacity to entirely offset their peak usage and sell back to the grid any excess then sure, seems like that wouldn't be a big societal net negative like it is right now.

If you care about CO2 then you do a carbon tax or similar (and then refund all the money to the public as checks so it doesn't damage the economy), at which point that's exactly what happens, only it happens for everybody and not just Bitcoin miners, which is what you want anyway.

You can also increase economies of scale by building out solar farms, and using them for something useful, instead of wasting it on guessing random hashes.

Saying that wasting energy is fine as long as you get it cleanly doesn't change the fact that you're still wasting it.

One thing we do not lack is demand for more energy.

Please tell me where I can get unlimited solar panels for free. I'll rent a truck and be there straight away.

> One thing we do not lack is demand for more energy.

Market demand is the willingness and ability to pay money for something. If the demand was actually unlimited then why isn't there either a Dyson sphere around the sun already or a 0% unemployment rate from everyone having a job building one?

As a non-American, it's hard not to notice that it's not future generations. It's everyone using dollars.

And since your country will be invaded if you try not using dollars to trade oil, and everyone needs oil (transport, food/fertilizers, medicine synthesis), then it's literally the whole world paying.

Which incentives USA to print money, because they only shoulder a small part of that burden.

Isn’t that exactly the point? Bitcoin incentivized wasting resources. It is, according to your own comment, unnecessary to use so much computing to keep bitcoin going. But it’s being used.

If Bitcoin were worth much less the network would still be secure even though the mining reward would only be enough to pay for a fraction of the current processing.

If Bitcoin does not double in value every four years, the mining reward will reduce in real world terms.

Claiming the mining resources required will be at the current level or higher perpetually requires also making the claim that you think that the value will increase exponentially forever.

Nothing increases exponentially forever.

It is an object store called Didgets (i.e. Data Widgets). Each Didget has a specific type. One type is used to hold unstructured data like a file does. These Didgets are unsurprisingly called File Didgets. Other types of Didgets can hold lists (used to create hierarchical folders, music play lists, photo albums, etc.).

Others hold sets of Key/Value pairs which are used to create a tagging system for other Didgets or columns in a relational table.

Using a variety of Didgets, I have been able to create hierarchical file systems where a simple query can find one or thousands of files instantly out of 200 million+ based on the values of any tags attached.

In the same container (called a pod), it can store tens of thousands of relational tables; each one capable of having 100,000+ columns and billions of rows.

The system is 'multi-model' so it could manage hierarchical data, relational data, graph data, or anything managed by a NoSQL system.

It is not only versatile, but is incredibly fast.

Microsoft is just one of the companies that routinely does stuff like that.

NIMBY causes energy prices to go up in areas that won't allow drilling, refining, nuclear or nat gas development, or power lines. When will the same happen for things like AI services?

Software has an amazing multiplier effect. It can be copied to millions of machines and run billions of times each day. Code that wastes resources (time, memory, disk space, electricity, etc.) can become incredibly expensive to run, even if it was vibe coded in a day for a few dollars.

Has anyone taken a serious look at all the code being spit out by AI with regards to how efficient it is?

To give you a concrete examples. Recently pretext library made waves. I looked at the code and noticed that isCJK could possibly faster.

So I spent 30 minutes TELLING claude to write a benchmark and implement several different, hopefully faster, versions. Some claude came up with by itself and some were based on my guidance.

You can see the result here: https://github.com/chenglou/pretext/issues/2

The original isCJK, also written by AI (I assume), was fast. It wasn't obviously slow like lots of human JavaScript code I see.

Claude did implement a faster version.

Could I do the same thing (write multiple implementations and benchmark them) without Claude? Yes.

Would I do it? Probably not. It would take significantly longer than 30 min. and I don't have that much time to spend on isCJK.

Would I achieve as good result? Probably no. The big win came from replacing for .. of with regular for loop. Something that didn't occur to me but Claude did it because I instructed it to "come up with ideas to speed it up". I'm an expert in writing fast code but I don't know everything and I all good ideas. AI knows everything, you just need to poke it the right way.

But it seems to me that there are many projects out there like mine. You start building something because it scratches an itch you have. You think it would be fun to build. You keep adding features and fine tuning the code because you want to see something work better and/or faster than anything else.

Then one day you look at it and say: "I wonder if other people will think this thing is as useful as I do (and be willing to pay something for it)?"

It might still be a work in progress, but it does a number of very useful things, so you now have to put on your marketing hat or team up with someone who is good at that.