Evigtech @evigtech

Hi Friends, I need your HELP. My technical books - System Design Interview Vol 1, Vol 2, and Machine Learning System Design Interview - are being illegally sold on Amazon .in (Amazon India). I’ve been battling this issue for months. My publisher and I have filed dozens of copyright claims, conducted test purchases, and collected evidence - but the problem persists. I feel lost and helpless. 100% of my books sold on Amazon .in (Amazon India) are pirated copies. This means: - More and more developers are receiving low-quality, pirated books - The pirates profit off my hard work - My books are getting hundreds of incorrect 1-star reviews due to piracy The pirated books are linked to my Amazon listings, so reviews meant for the pirates end up on my product pages. If you are in India, the books are now exclusively sold through Shroff Publishers. You can help by: - Sharing the post to spread awareness as widely as possible - If you work at Amazon, please help escalate the issue This issue means so much to me as an author. Thank you for reading and for your help.

121 761 2K 327K 268

View Details

System Design Codex @ProgressiveCod2

3 years ago

Scheduling Cron Jobs sounds easy. All you need is a script and schedule. And you are good to go! But imagine doing it at the scale of a product like Slack. Slack is a messaging platform for team collaboration. Its success depends on the right message reaching the right person on time. A lot of this critical functionality such as timely reminders, email notifications & message notifications relies on Cron jobs. However, as Slack’s user base and functionality footprint expanded, there has been a huge growth in: - The number of cron scripts - Amount of data processed by those scripts. This led to a dip in the reliability of the overall execution environment. Hence, it became imperative for Slack to redesign its Job Scheduling System. ✅ Slack’s Issues with Cron Execution - A single node executed all the scripts locally. It kept a copy of the scripts and one crontab file with the schedules. At scale, this solution wasn’t easy to maintain. - Vertical scaling of this single node to support a higher number of cron scripts became cost-ineffective - The individual node was a single point of failure. Any configuration issues could bring down critical Slack functionality. To solve these issues, it was decided to build a more reliable and scalable cron execution environment. ✅ The New Cron Execution System There were 3 main components of the new Cron execution environment. - Scheduled Job Conductor - Job Queue - Vitess DB Here’s what it looks like: Let’s understand the role of each part: 👉 Scheduled Job Conductor New service written in Go and deployed on Bedrock (Slack’s in-house wrapper around Kubernetes) The service basically mimics the cron behavior by using a Go-based cron library. The job of the service is to accept incoming job requests and hand them over to the Job Queue. It can run on multiple pods (1 Leader and others Standby) so that there is no single-point of failure. Here’s what it looks like: 👉 Slack Job Queue This was an existing component at Slack. Basically, the Job Queue is an asynchronous compute platform that runs about 9 billion pieces of work per day and consists of multiple queues. Here’s what it looks like internally: The queues are like logical pathways to move jobs through Kafka into Redis where the job metadata is stored. From Redis, the job is finally handed over to a job worker (a node that actually executes the cron job) 👉 Vitess Database Table Lastly, the Slack team used a Vitess table to handle the job data. The purpose of the table was to: - Handle deduplication - Report job tracking to internal users For those of you who may not be aware, Vitess is a scalable MySQL-compatible Cloud-Native database. In the new system, each job execution is recorded as a new row in a table. Also, the job’s status is updated as it moves through various stages (enqueued, in progress, done). Check the below diagram: Overall, the new service for executing cron scripts at Slack has made the process more reliable, scalable, and user-friendly === That’s all for now! If you enjoyed this dive into scaling Cron jobs at Slack, don’t forget to: - Destroy the LIKE button - REPOST so that everyone can learn from Slack’s Cron Job Execution process - BOOKMARK for future reference - Follow me for more posts like this. P.S. This post is inspired by the explanation provided on the Slack Engineering Blog (search for "Executing Cron Scripts Reliably At Scale"). Diagrams have been redrawn for clarity