The Wind Rises is my favorite Ghibli, something about it just draws me in like no other. Don't get me wrong, I love Howl's Moving Castle, Princess Mononoke and Spirited Away like the next person, but The Wind Rises is special to me.
I mostly agree with the author - Christmas tree is a mess. But I also think the final theme was too minimalistic. I find it incredibly helpful when language keywords like "try", "await", or "new" are highlighted.
I agree. Not even highlighting keywords is silly. The overly colourful example he gave is bad, but his alternative is bad in the opposite direction. Middle ground is where it's at.
Similar. But this looks like it requires generating explicit migrations. The equivalent would be if every time you wanted to make a change to your terraform, you had to plan and save the generated plan file to your repository.
Still not quite the right workflow IMO. I think TF nails it and that SQL things are held back by legacy thinking in the space.
Ahh, so they do! Their quick start link took me to their "versioned" workflow which is...basically the same thing any of a dozen tools has done for decades. Strange landing page choice to funnel into your market equivalence rather than differentiator.
Great question! We have retry logic in place to handle Postgres restarts. If the failure occurs in-place, you should be fine as the slot will persist. If Postgres fails over to the standby, PeerDB will wait until the slot is created. We did consider automatically creating the slot if it doesn't exist on retries, but ensuring data reliability (creating the slot right after failover without data being ingested) is tricky. So, as of now, we leave it to the user.
Thanks! I'd say, this is a very complicated problem left to be solved by user. Debezium also does not solve the problem it, and from DBA point of view it is a blocker for adoption. Would be nice to have some solution that would ensure that logical replication slot persists through failover.
Things got finally a bit easier with Postgres 16, which allows you to create replication slots on stand-by servers. It still requires manual intervention to make sure they don't fall too far behind their primary counterpart and to do the failover, but it's not too terrible either and it's guaranteed that failover can happen without missing any events. I've described the approach here: https://www.decodable.co/blog/logical-replication-from-postg.... Postgres 17 should make this even simpler.
I checked the code and noticed some things that you might want to address in the future (I've done major version upgrades via logical replication myself several times using a script similar to this).
The "default" way of bringing the target DB in sync with the source one (CREATE PUBLICATION on the source then CREATE SUBSCRIPTION on the target) only works for moderately sized databases. Even on ~50GB I noticed that it may take hours to catch up with the origin. There are a couple of solutions:
1. Drop all indices except for the primary keys before performing initial copy, and then restore the indices after all tables have been copied and replication stream started.
2. Instead of copying into a blank database, you could instead create a logical dump using pg_dump with snapshot isolation, restore it on the target DB using pg_restore, create a subscription in the disabled state, advance the logical replication slot to the LSN with which you created the logical dump, and then enable the subscriptions. This way your target database would only have to process the delta between the time you created a dump and the current time. I have a snippet illustrating how we did it: https://gist.github.com/take-five/1dab3a99c8636a93fc69f36ff9....
3. We found out that pg_dump + pg_restore is still very slow on larger databases (~1TB) and it's untenable to wait several hours. Remember that while you're waiting until the dump is being restored, the source database is accumulating WALs which it should later send to the target DB. If left unchecked for 12-18 hours, it could lead to running out of disk space. This was unacceptable for us, so instead of creating a target DB from a logical dump, we created a copy of the source database from AWS RDS snapshot, upgraded it using pg_upgrade and then set up logical replication (similar to how it's described in Percona blog: https://www.percona.com/blog/postgresql-logical-replication-...). Something like this can probably be achieved with ZFS snapshots.
--
Otherwise, it was very similar to how you programmed your library.
Recently, I saw an article here on HN on the same topic of migrating Postgres databases using logical replication. There was a very nice idea to set up _reverse_ logical replication after switchover so that it's possible to roll back quickly in case something breaks.
These are great shouts! Thank you so much for sharing. One of the operations I ran with this was ~500GB and yeah it takes a few hours to catchup. For us, it wasn't an issue, especially also since we were on AWS Aurora. However, that said, I can totally see it being not feasible on other database engine/systems. I considered the idea of dropping all the indices and re-creating it once the catch up is complete. I'd like pg_easy_replicate it to support that. It should make the initial COPY lot more swift. On a 200GB large DB it cut down down the total time by 90%.
Hearthstone has exactly this system (only friends can chat with each other). What happens quite often after some matches:
1. You receive a friend request from your opponent
2. If you accept it, they probably are going to send you a bunch of insults and then unfriend you.
We also started with "temporal_tables" C-extension, and also had to switch to NearForm's version written in PL/pgsql to be able to migrate our database to RDS.
Didn't notice any performance drop.
It also much easier to install (just run an SQL-script, no need to restart the DB) and it works everywhere, on all PostgreSQL versions. Great piece of work!
What about "The Wind Rises"?