I find this infra perspective so fascinating, as a career-long product platform engineer and solidly staff+. I don't argue that sean's writing (which is amazing) is a little overly mercenary in framing, but I'll pick a few choice sections I don't agree with here too. tldr I think the "product cares about speed; infra cares about leverage" staff engineer archetype is a false dichotomy we shouldn't encourage.
"In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts." -- I disagree that speed is the ultimate currency! A great product org also respects long term leverage, it's just _always_ harder to argue for. But it's best to build a strong portfolio of going fast (where needed), going slow (where high leverage), and if everyone agrees your "going slow" led to huge returns you get the best of all worlds. Frankly it's a sign of a relatively junior product engineer if they are myopically focused on speed at the cost of all else.
"But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons." -- Extremely true, and this is _equally or more true_ in product domains. My most valuable contributions have come from sitting in a product area long enough to generalize 5 micro optimizations into the macro engineering leverage we needed to drive an order of magnitude more value from the same engineering input.
"For some engineers, navigating this [high visibility driven] chaos is a thrill. For those of us who care about system stability, it feels like a trap." -- my protip for prospective staff engineers is to _never_ say you only care about [speed, stability]. In most cases you must care about both, and it's worth advertising yourself as such. If you self select out of companies that only care about [pure stability/pure ship velocity] there should be a valuable balance to strike and staff engineers are in a unique place to enshrine that balance in engineered systems.
"In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers." -- surely we can agree impressing customer stakeholders is even more important in (healthy) product orgs :) But it's a curious claim!
Great article, wonderful to hear more nuanced and deep discussion of the practice of extremely senior IC engineering. Kudos!
Firstly, thanks for the thoughtful comments across the post. Really happy you found it insightful!
> But it's best to build a strong portfolio of going fast (where needed), going slow (where high leverage), and if everyone agrees your "going slow" led to huge returns you get the best of all worlds.
I guess maybe I oversold this a little in the post but I definitely think that product orgs value speed more than infra orgs: there's often an mismatch when I speak to product engineers on how fast they expect us to build things for them vs how fast we can actually go while considering all our other customers and not breaking other usecases.
> my protip for prospective staff engineers is to _never_ say you only care about [speed, stability]
Fully agreed, both are important for engineers to have. As above, I think the relative composition of them varies though depending on the area (just pulling numbers from the top of my head, maybe it's 70/30 in favor of speed in product orgs, 30/70 in favor of stability in infra orgs).
> surely we can agree impressing customer stakeholders is even more important in (healthy) product orgs
You're absolutely right for what I say in the post; reflecting I think I maybe did not go into this topic in enough detail for the nuance it deserves (but the post already felt long enough!).
Let's split the discussion into 3 different areas (as IMO they each work slightly different): infra/devex, B2C and B2B.
* Infra/devex: as I say in the post, it's critical to impress your other team's managers as that's how you prove impact.
* B2C: your customers are consumers so it's all MAUs, revenue etc. There is no "customer stakeholder" who will give you direct feedback for your promo packet
* B2B: here's where it gets interesting. So if your management chain is directly able to talk to customers (and especially senior managers) to solicit feedback without middle layers (parter manager, account managers, PMs) then yes you're absolutely correct. But often this is not the case: the middle layers act as a filtering point so you get only a fuzzy sense of how stakeholders in the other company about a specific technical thing you worked on. So again it's sort of at the whim of how your manager feels the other company's managers feels.
The basic point I was trying to make is that if you're working on external facing product, from my understanding, even if you impress your external stakeholders, it's not like in your promo packet you can attribute concrete quotes to the customers. quotes like "we couldn't have done X in our company without the work of Y engineer in your org who worked on Z". Whereas this sort of quote is extremely common to see in infra promo packets.
Hope this made sense, I'm not sure I communicated this last point well!
Finally, I cannot wait! This sure sounds like it should remove the terrifying guesswork of wondering what the actual quota values are, detecting when you're approaching a limit, and starting to automate or at least alleviate the pain of changing them.
Judging just by the blog post, this might well be the best quality of life improvement of 2019.
In the console, I see at least some services (I care about Step Functions) list as "showing default quotas only". I wonder if this means there will be a long tail of adoption before individual services show actual current values?
Strong agree -- the distribution was a solid deal when it happened and didn't require a paywall. I just gave up on them (http://www.locallyoptimal.com/blog/2019/03/24/publish-indepe...) now that they're forcing you to choose no-paywall or medium distribution. If I'm not wanting a paywall I may as well use my own site and be totally free.
It's a shame, I found the editing interface pretty convenient (image embedding in particular) and conceptually the Medium-run publications make a ton of sense.
Inherited a codebase that ran nearly all revenue-critical operations, and operated at its core on some of the most metaclassy/dynamic tools Python has at its disposal.
Luckily I work in a tech company where the fact that nobody knew this code and it took years to effectively ramp up on was argument enough that it should go away.
Actual removal was a much messier story. It had tangled deeply into adjacent systems so you couldn't "just replace it". We are in the later phases of something like the Strangler Pattern (https://docs.microsoft.com/en-us/azure/architecture/patterns...) where we built higher-level interfaces over the top and gradually re-implemented the underlying functionality without using any of these custom frameworks.
That said, it's a long term project that is easy to lose steam on. It's been very important to regularly revisit our goals and how we're attacking them...AWS has released services that fundamentally changed our approach (for the better) in the years since this effort started and we've probably cut off at least a year from the overall effort by adopting those instead of continuing on the original course.
I wrote up some of these ideas about accomplishing big projects that span years at https://medium.com/@scott_triglia/ask-a-tech-lead-i-have-to-.... The parts about regularly re-evaluating the next steps in your course of action were directly inspired by this project I just described.
Great to see open source tooling around Step Functions. Personally I'm still holding out for someone taking the States language spec (https://states-language.net/spec.html) and turning out higher-level tooling for writing/inspecting/manipulating state machines.
Agree w/ bpicolo (hi Ben!) that the tooling still can come a long way for those who can't use Lambdas directly. This code is interesting, but it misses my major use case for Step Functions -- running a workflow across several services transparently.
If I had to guess the future, I suspect that tools like Step Functions will become more and more crucial as we realize that in a FaaS world the job of coordination is complex and essential.
What makes you say Step Functions isn't ready for prime time? We've been using it (and SWF which it is based off) for about a year at decent scale and been generally very happy with it.
When I initially set the up, there was no way to edit or delete them. I haven't messed with them since, although, I did just check and there aren't any step functions in my account so it looks like they deleted the old ones.
It may be time to test them out again, I just go bit pretty bad with the last time I implemented them and lost about a weeks worth of work because of it being un-usable.
So I ask this mostly out of horrified curiosity -- is there any negative press or revelation that could actually sink Uber at this point? What would it take for the company to fail?
We've seen:
* Systematic flouting of laws in public
* Even further hidden avoiding of laws
* Sexism complaints across the board
* What appears to be a pervasive toxic culture, inspired (if not explicitly encouraged) straight from the CEO's behavior
They're trying to become too big to fail, but aren't quite there yet.
Essentially they're undercutting all competitors by artificially keeping fares very low and subsidizing drivers. Subsidies were the primary source of their enormous financial losses in 2016.
Once the competition is completely gone (local cabs, and Lyft in some areas), then they can stop subsidizing themselves and basically hold a monopoly. It's their entire strategy.
I simply don't see how Uber can be a monopoly. You need a critical mass of drivers, but nothing stops drivers from being on multiple networks. This isn't Microsoft doing per-CPU licensing.
> I simply don't see how Uber can be a monopoly. You need a critical mass of drivers, but nothing stops drivers from being on multiple networks. This isn't Microsoft doing per-CPU licensing.
They could force drivers to only work for Uber. "If you go to the competition, you can't work for us", they could try to lobby so that other networks can't legally operate, just like Taxi companies did... They just want to replace Taxi companies, that's why people who think they are the "hero of the libertarians" are misguided.
They could keep track of people as best they can, and shadowban them from all transportation using Greyball so the people couldn't function in society. It'd be kind of like getting blackballed for life from credit card issuers. Effective to the extent they can make a monopoly, probably more effective when it's a shadowban rather than explicit rules stating the person is barred from using the service forever.
It would be rather profitable to be able to hold the world hostage, single out whoever you want and make them hoof it. Particularly if they'd got used to using Uber to get to work or something. Long game would be making private vehicle ownership illegal, but that's very long term indeed. I'm just saying, you can keep going and every step of the way is increasingly profitable. Wall Street would love it.
While I agree with your first statement, the per-CPU analogy may not be right. The CPU (or car) is the resource being leveraged by the OS (or ride hailing service). Also, I may be wrong here but I think it's per motherboard.
For Uber to really become a monopoly would be hard, at best they could split the field with Lyft or others by making an under the table anti-"employee"-sharing agreement similar to the Apple-Google thing last year.
That and about 98% of the USA hasn't heard about any of these bad news stories nor do they care. If their rides start to take 30 minutes to arrive then people will start to care which is what matters. I think much of the rest is Silicon Valley echo chamber stuff.
More or less Redbox' strategy too, from what I can tell. Prices have been creeping up ever since the major competition (Blockbuster et al) dropped out of the market. And I suspect it's not because the costs to run machines has been increasing...
I imagine a great deal of it is the studios increasing their take. This has been a perennial problem for Spotify and other licensees of other media content. Netflix rolls their own now.
Really, it's hard to imagine a future for Redbox in ten years. Even the convenience of a vending machine movie store is nothing next to me not even getting off the couch.
With the prices at $2, which becomes $4 if I don't remember to return it on time... I often opt to pay the $5 to rent using a digital platform. Only time I pick up from Redbox is if I am at the store anyway.
I read a relevant thesis about the timing, structure of presidential scandals. Written by Daniel Ellsberg of the Pentagon Papers fame? (Sorry, can't refind it quickly.)
The gist is that scandalous behavior doesn't matter until it does. All the negative stuff everyone already knows about a protagonist is discounted, for a time. And then a switch is flipped and everyone piles on.
One example given was Clinton's sexual indiscretions. It was an open secret. It only became a "scandal" when opinion shifted. (For whatever reason.) Then it's unshakeable.
Not even close. If Uber disappeared tomorrow it would have literally zero impact. You'd get the same driver in the same car via another app and one ride later you'd have forgotten Uber ever existed.
They will succeed or fail based upon their ability to attract customers and sell a product at a profit. I suspect most customers don't care about any of your bullet points. They care if Uber can get them where they want to go faster and/or cheaper than other options.
What I have noticed is that even people that have concerns about their culture and unethical acts, will often use the service because it's convenient and (for now) cheaper than other alternatives.
As long as Uber is fighting cities (instead of state/federal governments) and continues to have deep pockets... They will be around.
Seems like they haven't considered the fact that they would then be in the same situation their current competitors find themselves in, ripe for "disruption."
I think this probably counts as an iceberg-scale problem. It's like taxes; avoiding them by gaming the system in every way you can find is unethical but not illegal. Actively deceiving people (with fake updates to the app) and collecting and acting upon PII to screen out public officials is getting into criminal conspiracy territory.
I'm extremely surprised their lawyers signed off on it. Maybe they're libertarians though, I've heard some really odd arguments that turned out to be predicated on various 'natural law' theories.
Very much looking forward to an upcoming project with Step Functions at $DAYJOB. I suspect there are many use cases for turning complicated intersections of batch jobs and blobs of code into discrete, independent tasks like SFN requires.
Particularly interested in composing higher and lower level state machines for better separation of concerns, but haven't yet come up with a compelling use case :)
Is it possible to accept an activity task with no timeout, store the state in a DB, then when your event occurs, load state from the DB and send the task_success signal? Not convenient, but seems feasible..
As someone currently at the "prove it's a good idea" stage of moving production from SWF to SFN, I'm happy to hear it's been such a good experience in practice!
I think I'm mostly excited to stop locally representing/maintaining/enforcing the state machine -- it's been a huge source of toil for very little benefit. Hoping the cheapness of a new state machine will encourage splitting them when it makes sense instead of inflating/complicating an existing workflow...
How are the latency gains in practice? Specifically the activity -> decision -> activity interstitial latency in SFN vs the old SWF style.
I haven't actually worked with activities in SFN, just the pure Lambda state machines. But I can tell you those are fast. Like single-digit millisecond latency between states fast.
"In the product environments Sean describes, where goals pivot quarterly and features are often experimental, speed is the ultimate currency. You need to ship, iterate, and often move on before the market shifts." -- I disagree that speed is the ultimate currency! A great product org also respects long term leverage, it's just _always_ harder to argue for. But it's best to build a strong portfolio of going fast (where needed), going slow (where high leverage), and if everyone agrees your "going slow" led to huge returns you get the best of all worlds. Frankly it's a sign of a relatively junior product engineer if they are myopically focused on speed at the cost of all else.
"But the more powerful return is systemic innovation. If you rotate teams every year, you are limited to solving acute bugs that are visible right now. Some problems, however, only reveal their shape over long horizons." -- Extremely true, and this is _equally or more true_ in product domains. My most valuable contributions have come from sitting in a product area long enough to generalize 5 micro optimizations into the macro engineering leverage we needed to drive an order of magnitude more value from the same engineering input.
"For some engineers, navigating this [high visibility driven] chaos is a thrill. For those of us who care about system stability, it feels like a trap." -- my protip for prospective staff engineers is to _never_ say you only care about [speed, stability]. In most cases you must care about both, and it's worth advertising yourself as such. If you self select out of companies that only care about [pure stability/pure ship velocity] there should be a valuable balance to strike and staff engineers are in a unique place to enshrine that balance in engineered systems.
"In a product organization, you often need to impress your manager’s manager. In an infrastructure organization, you need to impress your customers’ managers." -- surely we can agree impressing customer stakeholders is even more important in (healthy) product orgs :) But it's a curious claim!
Great article, wonderful to hear more nuanced and deep discussion of the practice of extremely senior IC engineering. Kudos!