@Mer__edith upfront: i do highly value your work, both personally and that of the whole signal team. you are doing a great service to society!
also, i have not run signal's infrastructure, but i have run ISPs until 2008 and am still helping various clients deliver their serices. i will
not mention any names, but you know many of them. i might know a little bit about scalable software… 🧵
Post
@Mer__edith
https://keet.io/
https://docs.pears.com/
uses only edge devices as infra , does voice / video / txt / application distribution via hash urls, solved multi device identity in a p2p space , has search , no limits on file sharing , scales like bitorrent can do ip obfuscation via blind peers, has a whole ass software ecosystem that grows by the day.
it has bugs but it is amazing how well it works and i would say it's ready for scale up.
@Mer__edith it's a good app to use but uploading and sharing a video is a problem because the quality of the video is low.
@Mer__edith It is not about accessing someone's data but about relaying on such infrastructure. Those are two different things. We have already great example how can it be and what happens if you aren't fully independent from such services. This should be a very good lesson for the future, for you, your company and all the Sigjal users as well. It is right about time to start going decentralized way, don't you think?
@Mer__edith stop using excuses. You are paying American companies that do not respect privacy. There are several European alternatives that you can choose from, especially Germans.
@Mer__edith
There are two sides of the same medal.
We cybersecueity specialists always rank hosted services on AWS on a higher rank than a self hosted or elsewhere hosted one. Always, no matter how safe it is.
The second thing is, no matter the risk of hosting on AWS, it is a correctly cooperation and support of Amazon, Amazon infrastructure, Amazon services and services companies.
This support is by promoting the service activity or passively, directly payment for a company which is officially support a president, his ICEs, his MAGAs and politics.
Our problem has NOTHING to do with the quality, security, E2EE of the Apps Services, Servers and Owner. Our problem can not be fixed by a simple update, but by MIGRATION.
And migrations is a main topic in the US.
@Mer__edith Thank you for taking the time to explain all this!
In a completely different direction -- are you actually on a server that limits you to 500 character posts? I mean, I am, but I'm not a communications expert.
@Mer__edith think the surprise is that many, perhaps naively, consider Signal to be part of the resistance against big tech oligarchy and have little understanding of how insidious Amazon is. It’s not like one can just boycott them out of existance by cancelling prime and not shopping on Amazon.
They are a perputual money machine.
@Mer__edith We may have to go back to paper and pencil!
My 2 cents:
Being on the Fediverse and also running an XMPP server, why do we use centralized systems?
Corporate takeover and control.
@Mer__edith Serious question: The concentration of providers for real-time, global networks, can it really be avoided? If we had ten of them instead of the , I don't know, the three? we have, they'd all have to have more or less the same kit, duplicating investments for a smaller share of the market. That would make it more expensive. Would that even work?
You can probably tell I'm not in this kind of business, but I'm a citizen of a European country, and de-coupling from US is dabated.
@Mer__edith upfront: i do highly value your work, both personally and that of the whole signal team. you are doing a great service to society!
also, i have not run signal's infrastructure, but i have run ISPs until 2008 and am still helping various clients deliver their serices. i will
not mention any names, but you know many of them. i might know a little bit about scalable software… 🧵
@Mer__edith that said, i think there is a real problem with the narrative of your thread, that the problem at hand was so big that it needed the help of a hyperscaler, that not even signal could run their own infra on bare metal.
in my world, clients who have infra in perfectly good shape, scaling out, performant, whatever, voluntarily give up this privilege and submit themselves to #capswag on the basis of anecdotal evidence like this… 🧵
@Mer__edith if you have, and i assume you do have, fully automated your ops, you need apis like k8s or s3, sure. but that is not the same as buying from a hyperscaler. you could run the platform yourselves, and maybe still use public cloud services for "spill over".
i apologise for the unsolicited advice, but i can not imagine that signal was not big enough an organisation to be able to run their own infra.
please do seriously consider this option. if not you, who should?
@Mer__edith for me, the AWS outage drew attention to Signal's centralized nature - that it runs on servers controlled by Signal and hosted in the US. Moving away from AWS would not address this problem. I have chosen to use decentralized messaging apps instead. They are just as secure, more resilient, and not dependent on Big Tech cloud infrastructure.
@Mer__edith It'd be really nice if I could run my own private signal server (or one server in a private cluster) so that an important part of my social network didn't *have* to depend on AWS.
@Mer__edith @Binder As I read through the thread my first thought was to convert these infrastructure systems into government-managed utilities. Of course, that would deter innovation and efficiency. Which leads me to development of utility regulations required for infrastructure systems allowing competition and profits, perhaps with some government investment equally spread across the competitive systems.
@Mer__edith Very insightful thread. Thank you for clearing this out. Unfortunately AWS is almost a necessity these days for scaling application
@Mer__edith Yeah and this is why when there's something with a use case similar to what #Signal is intended for, I always ask if it's #P2P.
@Mer__edith Thanks for the honest and thorough explanations! 🙏
While reading it, I was already afraid of the responses you would receive and they were partly confirmed.
Keep up the great work 👍
@Mer__edith how’s people comparing Signal to Mastodon? Or their own self-hosted service, software or whatever? How people cannot understand the privilege they have to know how and being able to run a self-hosted service?
@Mer__edith Thank you for the explanation. I support Signal.
This situation shows that more people should support EU companies whenever possible, rather than large corporations. We need alternatives for various services and must take small steps in this direction https://european-alternatives.eu/
@Mer__edith ,Wait, how does Threema work, for example?
@Mer__edith There are other options, others are doing it. Using AWS, GCP and Azure is a *choice*. You *can* technically build without them. It would also probably cost less. This is a political choice you are making.
@FreakyFwoof @Mer__edith I knew there was a reason I didn't ever use signal. lol
@Mer__edith
Thank you for the thread. It’s honest and straightforward, making it a perfect example of “privacy is a luxury” and not aligning with mass processing..
@Mer__edith
It is, to me, even more concerning how no one (none of our elected deciders anyway) seem to realise how powerful the #clubofoligarchs is. None of any nations matter at all when said club decides to (seriously) *use* that power.
Seem to be a different topic? Imho nope.
This AWS incident🤨 shows what *could* happen, giving just a glimpse of it.
looking at your replies to replies here that seem to make sense to me (especially re decentralization), you're telling them they don't know what they are talking about. well I definitely don't.
like with debates re #ATproto and #ActivityPub, I have thoughts but I know that we really need to see the experts debate each other somehow. I don't think it happens enough. so I'd say the same re #signal and #matrix etc.
@Mer__edith Thank you for layout this out and continuing to school us. What do you think is the major first solution to end this concentration in computing power - anti-monopoly reforms?
@Mer__edith interesting engagement levels across different sites.
3 media
@Mer__edith one question I had as an SRE, do you use the same kinds of services across all the three big cloud providers in a multi-cloud configuration, or do you use unique/distinct functionality from each?
@Mer__edith not so surprising, as it's very difficult to do anything at any scale online and avoid AWS entirely.
The surprise shouldn't be about Signal, it should be a rallying cry to build diverse infrastructure.
Concerning, bc it indicates that the extent of the concentration of power in the hands of a few hyperscalers is way less widely understood than I’d assumed. Which bodes poorly for our ability to craft reality-based strategies capable of contesting this concentration & solving the real problem. 2/
@Mer__edith That's only true for centralized models like Signal, which therefore depend on centrally-controlled infrastructures. My grandma was wiser: she told me not to put all the eggs in the same basket. Granted, she was not CEO of a billion $ company.
The question isn’t "why does Signal use AWS?" It’s to look at the infrastructural requirements of any global, real-time, mass comms platform and ask how it is that we got to a place where there’s no realistic alternative to AWS and the other hyperscalers. 3/
@Mer__edith I mean, you could host a distributed network of servers hosted in multiple different countries indifferent regions, it's just that it's more difficult and expensive than, you know, paying Amazon to do that for you.
We're a lazy species in general 6o my great disappointment.
@Mer__edith What if, instead of running a global comms platform for millions of people that requires AWS level infrastructure, we run a bunch of small, local ones that all federate and interop with each other? 😍
@daniel @Mer__edith Even _IF_ it were possible to create a black box version of "distributed Signal mesh node in a box" that you could run in your basement to help make Signal more tolerant - I mean with enough $ and willpower Im sure it could be done - there's still the question of: if you don't control physical access to the node, there's still potential for attack regardless of how much encryption and protection. Would you ever be able to trust it completely?
The key word is "infrastructure". Infrastructure and (so called) free market don't go together well.
Running a low-latency platform for instant comms capable of carrying millions of concurrent audio/video calls requires a pre-built, planet-spanning network of compute, storage and edge presence that requires constant maintenance, significant electricity and persistent attention and monitoring. 4/
@Mer__edith But if the choice is “having a low-latency messenger” or “having a mediocre-latency messenger, not being dependable upon Bezos”, I’d take the mediocre-latency messenger, tbh. I don’t need on-the-second arrival times of text messages.
Yes, I am only one guy, and the Signal userbase is huge. But what fits Signal’s mission and expectation of its users best? Being available when others are taken down, albeit taking a bit more time to send/get acknowledgement of read status? Being state-of-the-art-fast&lean?
This is not a yes/no/easy decision, I get it. But I still would prefer Signal not being hosted within the reach of a maniac, and its friend, another maniac. Not a matter of encryption, but both hold Signal’s power-off-buttons next to their thumbs.
@Mer__edith I don't understand the problem. You have commerce of the scale of the Romans because only the Romans have the brains, resources and reach to build the surprise surprise"Roman roads level commerce"
Oh shit, how do we do Roman roads level commerce without the roads? You don't, you stay in Feudal middleage localized commerce.
Instant messaging demands near-zero latency. Voice and video in particular require complex global signaling & regional relays to manage jitter and packet loss. These are things that AWS, Azure, and GCP provide at global scale that, practically speaking, others (in the western context) don’t. 5/
@Mer__edith Just like almost everyone else, Signal is using a WebRTC-based stack for calls. This means:
- Signalling is forwarding SDPs generated by libwebrtc to other client(s). This is just a few messages at the begin of a call, doesn't need to be low latency and can just use the regular message infrastructure.
- Actual data transfer uses direct p2p connection if possible or TURN if not. The TURN server can operate completely independent of the messaging infrastructure.
@Mer__edith Thus I'm a little confused why you bring up calls. While it's true that maintaining a good TURN service is a complex task, I fail to see why this could only be done on with the hyperscalers. Notably, TURN servers running in parallel don't need to know about each other at all and typically one call would use exactly one TURN server, meaning there's hardly any interaction between TURN server instances, allowing to run those in arbitrary many data centers with different providers.
1 more replies (not shown)
@Mer__edith does that mean that it could be made possible to fallback to a system where messages still get through, just not in real time?
This isn't ‘'renting a server.' It's leasing access to a whole sprawling, capital-intensive, technically-capable system that must be just as available in Cairo as in Capetown, just as functional in Bangkok as Berlin. Particularly given the high stakes use cases of many who rely on Signal. 6/
This isn't ‘'renting a server.' It's leasing access to a whole sprawling, capital-intensive, technically-capable system that must be just as available in Cairo as in Capetown, just as functional in Bangkok as Berlin. Particularly given the high stakes use cases of many who rely on Signal. 6/
@Mer__edith But you don't have to rent a server or such, if you manage to let the peers communicate directly with each others. There is still an extremely expensive and large amount of hardware behind it, it's called “the Internet”. But it's not owned by one rich Bezos guy.
Decentralized is clearly more difficult. But it means you don't depend on any special large service which can be forced by legal means to conform to whatever surveillance laws there are.
@Mer__edith amen, this is what I keep telling my clients as well: AWS gives you a standardized product that works reliably well at a small scale and at an insane scale.
I always give them the option to hire teams of ops people to maintain servers and infrastructure 24/7 too. But then the high capex is also quite off-putting.
Eventually for small companies (I usually deal with ~50 FTE companies providing a SaaS product), it just makes more sense to use AWS and focus on their core product instead.
