I happen to learn best using audio and thankfully AI is making it easy. I use NotebookLM to create audio overviews. This is a review of Course 1 & 2 of Andrew Ng’s ML specialization. Enjoy!

The other day, I was listening to a podcast and wanted to cast it to my Google Home. But, guess what happened? I couldn't figure out how to cast the podcast from the Apple podcast app to Google Home. 

What is the incentive for Apple to develop this capability? Google Home is not an Apple product and even worse these are rivaling ecosystems. Android phones also do not let you Airplay content to your Apple TV.

I realized that there are many podcast apps on the app store. I searched for one with this functionality and tried a couple. Finally found the one that worked (📢Pocketcasts)! It was such a pleasurable experience! 🙌🏼😄

I had another similar problem. I have Chromecast for my TV and I was unable to find a way to cast Apple TV to Chromecast because it only supported Apple’s Airplay. For this one though, I had no recourse. Except to play Apple TV on the Chrome Browser of my laptop and cast it that way.

This felt like a tale of two cities. On reflecting, I realized that the reason for such a different experience was STANDARDS. Yes, podcasts are usually distributed over the internet via RSS (Really Simple Syndication) feeds. RSS is a web syndication format, that specifies a standardized, computer-readable format for publishing and consuming content. That's why many applications can read and play podcasts. Videos, on the other hand, are not distributed over a standard. E.g. Youtube videos are only accessed via the Youtube app. The developers cannot build an app to play videos without using YouTube APIs (which might be hard to get access to).

Why should Software Engineers care?

As a software engineer, this realization can help in a few ways.

Develop products that adhere to standards

Let’s say you are designing a web service. You can design your interface using the OpenAPI specification. Since that is a widely expected standard, developers can develop tools for that specification and get broad adoption. E.g. automated client creation, and endpoint UIs for OpenAPI specifications. You can leverage these tools for your development.

Use products behind a standard API

Let’s say you want to make your systems observable. Instead of directly instrumenting monitoring system clients like Prometheus, Datadog, etc. You can choose to instrument your application with OpenTelemetry. You can then configure the backend to either of these monitoring systems. This enables you to easily switch backends in the future and gives you that optionality.

Develop/Improve tools for common standards

Since standards are shared across organizations. If you have a certain challenge you can solve it for everyone using that standard. E.g. If an email client, in your preferred language, is not performant and you have certain ideas. You can create a different client that follows the same SMTP or IMAP protocol. Your solution can then be used by everyone using that protocol.

Regulation & compliance

Understanding standards like HIPAA & GDPR is important to make your applications legally compliant.

Hope this helps you realize how important standards are. 

Please subscribe to my newsletter for more insightful articles.

Kubernetes has emerged as the de facto container orchestration platform. Its flexibility, scalability, and robust ecosystem make it an ideal choice for managing complex microservices architectures. Alongside Kubernetes, Infrastructure as Code (IaC) has revolutionized the way we manage infrastructure resources, providing a more reliable, reproducible, and version-controlled approach to deployments.

Terraform is a popular IaC framework with support for most cloud providers. Kubernetes also provides a mechanism to extend itself to manage custom infrastructure (resources) via Custom Resource Definitions (CRD). The wide adoption of Kubernetes as a computing platform leads to the question of using CRDs as an IaC tool. In this post, we compare both these tools and identify the use cases fit for them.

Developer Experience

Terraform

HCL (HashiCorp Configuration Language) is the configuration syntax used by Terraform to define and manage infrastructure resources. You can extend the framework by writing modules to manage custom resources. You can choose the backend to allow Terraform to store the state. I faced some problems when my dev environment became out of sync with the actual resources, but eventually found my way.

Kubernetes CRD

The resources would be expressed in YAML. For functionality like loops, you would need a templating engine like Jinja. If your services are already deployed on Kubernetes, this is a good option for managing cloud infrastructure the same way. This is also a good option for custom resources. E.g if you wish to manage custom Kafka installation, you can use Strimzi.
One more advantage of using CRDs, is that they provide a RESTFul interface to manage your custom resources. This enables more ways to manage your infrastructure.

Adoption & support

Terraform

Terraform is quite mature, so most cloud providers already have terraform modules. You can check out the list here. The codebase seems pretty active & there is a lot of support on Stackoverflow.

Kubernetes CRD

Kubernetes is itself a very mature project, but the operator (CRD) implementation might not be very mature. There is an open-source project Crossplane, that provides CRDs for all cloud providers that support Terraform. AWS & Azure have open source providers (CRD). Adoption & support wise Terraform is more prevalent, but Kubernetes CRDs have a good enough ecosystem to make it useful. If your organization is already a Kubernetes shop, it is worth considering using CRDs.

If your org is heavily invested in Kubernetes, CRDs may integrate seamlessly, while Terraform offers broader support for diverse infrastructures. Assessing the complexity of managed resources & technical skillset can guide your decision.

Broadly, there are two ways in which metrics can be collected and monitored.

Pre-aggregated metrics

In this approach, metrics are explicitly instrumented in the code. The metrics are collected and/or aggregated into a time-series database. Examples of these are Datadog[2] and Prometheus. 

Since the database is designed for metrics, it is very efficient and cheap. The solutions also come with rich statistical functions, so it's easier to query.

The challenge is to know and explicitly instrument all the useful metrics for your monitors.

Log-based Metrics

In this approach, the system is designed to store logs but does have a query language to analyze and visualize them. Examples of these are Splunk, AWS Cloudwatch Log Insights, and metric filters.

Log-based monitors can help capture unknown metrics that were not explicitly instrumented. Also, since logs are fundamental for any observability, log-based monitors are easier to get started with.

However, there are some challenges like queries can be complicated and brittle. A seemingly harmless change in a log statement can break the monitoring. Besides, the performance of log-based metrics might not be great because the db is not designed to compute metrics and there is no pre-aggregation.

How to use these systems?

If you have a fairly small use case, log-based monitors can be a great place to start.

However, as you scale, it would make sense to incorporate both of these solutions to set up your metrics monitoring.

Recommended strategy

• Use a pre-aggregated system to monitor core metrics like Request counts, 5XX counts, percentile latency, and any other useful custom metrics.

• Use log-based metric monitors as catchall for anything in the log that is unexpected. E.g. Number of warn logs, error logs, or any other generic suspicious patterns.

The above strategy would help in achieving great monitoring coverage, while not sacrificing efficiency and cost.

Hope this helps in developing a framework for different metric systems.

References

[1] https://www.dynatrace.com/news/blog/what-is-observability-2/

[2] https://www.infoq.com/presentations/datadog-metrics-db/

Code inspection/analysis

On the top bar : Code → Inspect Code

This feature enables you to run built-in static code analysis on a selected scope or directory. It compiles a list of all the identified issues and links to the relevant code sections. What's particularly useful is its ability to streamline simple refactors, such as removing unused imports. It proves invaluable for tidying up legacy code bases.

Verson Control System (Git)

Location: On the top bar : VCS

When I started using IntelliJ, I initially relied on the terminal for Git operations. However, I was pleasantly surprised to discover that IntelliJ offers excellent support for version control. The commit section near the project displays all uncommitted files, allowing for a quick review before committing and sending a merge request. Additionally, IntelliJ provides commit message editors and templates.

One particularly noteworthy feature is the Git tab on the bottom panel, which offers a user-friendly interface for committing changes, undoing commits, reverting changes, and modifying commit messages.

My personal favorite is the merge conflict resolution tool, which enhances the user experience by providing a clear view of all changes and simplifying the merging process. I highly recommend giving it a try.

Database consoles

Location: “Database“ section on the side panel

For the longest time, I didn't realize that IntelliJ could interact with SQL databases. This functionality proves valuable for querying databases within the development context, minimizing the need for context switching. Additionally, it offers code completion for SQL databases and allows for the exploration of database tables and columns easily.

These were some of my favorite but underrated features. If there are others that I've overlooked, please feel free to let me know—I'm always eager to learn.

The good parts

Copilot is extremely fast in coming up with multiple code complete suggestions. Around 30% of the times I accepted the suggestions without any modifications. It has a decent grasp of the context of your code, like the code above and below, opened files etc. It learns from the way you code as well. If I do not take Copilot’s suggestion and instead do something new, it realizes it was wrong and plays along well.

Also, it reduces the number of my keystrokes, which is incredibly useful for people with slower typing speed (including me). I have been using all the IDE features like version control UI, code generation, run configurations etc. and this is a boon for me.

The challenges

I am used to using classic IDE auto-complete suggestions. They are often correct because those are based on actual classes and compiled code. Also, the IDE code generation capabilities and refactoring rules ensure that the code before and after is logically equivalent. I found Copilot to be making up stuff a lot of times. Since I have the muscle memory of using the suggestions as is (thanks to awesome IDE code suggestions) I often found myself accepting sub-optimal or flat-out incorrect code from Copilot. Changing it back again and fixing it was a little frustrating and I wondered if I could have written it better myself. Also, I missed correcting or verifying the Copilot generated code a couple of times, which was scary. We know that Large Language Models (LLMs) often hallucinate, and Copilot is based on one such LLM (OpenAI Codex), it’s hard to be mindful of that while coding.

Another challenge is that it does not usually follow the best practices or the rather the practices that are consistent with the code base. E.g., Using for loops vs the lambda functions. This may or may not be a big deal, but since new developers would be relying on such tools, they need to be careful.

I also often found the Copilot to be glitchy, I am sure this would be resolved as the service matures.

Last weekend, while listening to The Innovator’s Dilemma, the classic book on the nature of technological disruption, I was thinking about its parallels with writing clean code. The core idea of the book is that established companies focus too much on the needs of the current customers and may miss out on disruptive new technology. Startups on the other hand do not have that ‘baggage’ and can disrupt the incumbent’s business with innovation. The dilemma then is for a company to balance the need of current customers with the need to innovate. 

Surprisingly, I experienced a similar dilemma while struggling to prioritize cleaning up an existing code base and delivering direct end-user value with feature development. Focusing on current customer needs seems to be the right thing to do, but it comes with the cost of slower feature development over the long term because of poor code quality. Besides, it is even hard to convince the product management of the value, given the incentives are to deliver end-user value faster. Ultimately, a lot of projects are ‘disrupted’ by new initiatives to rewrite the code base or by a SAAS offering. Both of the alternatives are expensive. 

Writing good code is easier in the earlier stages of the project, just as innovating is easier for a startup. 

Interstingly, innovating and writing quality code both get progressively difficult over time. Since the code after a while solves existing use-cases, it is hard to remove/refactor a bad interface and improve the code structure. Just like an established company has a ‘baggage’ of existing users, code has the ‘baggage’ of existing code, features, and use cases. 

Prioritizing between refactoring and feature development is hard. One thing we can all do is keep the campground clean, i.e. leave the codebase in a better state than it was found.

Law of leaky abstractions

The law of leaky abstractions is a principle in software engineering that states that all non-trivial abstractions are leaky. It is impossible to create perfect abstractions. Even if your software is perfect it is not immune to natural disasters, electric grid failures, etc. However, there are reasonable expectations of abstractions that users expect out of the system. E.g. Useful error messages on failure, reasonable performance, and correctness.

Programming Model

Spark started as a platform that was a replacement for Hadoop MapReduce. It solved problems of disk-based data exchange, lack of expressivity, and fault tolerance without data redundancy¹. The solution was an abstraction called Resilient Distributed Dataset (RDD), an in-memory, distributed (partitioned across machines), immutable collection of records.

RDDs were designed to implement common functional operations (e.g., map, reduce, filter, etc.) on collections. It’s easy to think about them as any other collection (List, Array).

E.g., To check the emptiness of a collection in Scala. myList.size == 0 and myList.isEmpty would perform similarly. But rdd.count() ==0, and rdd.isEmpty would not. Spark counts the size of rdd, there is no pre-computed value.

This is a leaky abstraction because rdd.count() ==0 and rdd.isEmpty is logically equivalent and the expectation is that performance would be comparable. Spark’s

As the project evolved, the Dataframe API was created, which paved the way for Spark SQL. Spark SQL abstracts away many details, but it has similar problems. Here is my previous article with an example of equivalent queries with different performance.

Configuration

Configuring a Spark job can be complex due to numerous configurations. Typically, configuring memory on worker JVMs (Executors) and the number of tasks per executor is necessary.

Optimal values for the above configs depend on

• The Code being executed

• Number of i/p files (aka partitions)

• Size of files (aka size of data)

• Type of file format (CSV, JSON, Parquet)

• Data distribution

In addition, there are other non-trivial considerations like compression, serialization, and observability.

Spark does not hide the complexity of understanding and optimizing these configurations. Developers must at least have a mental model of the Spark internals to utilize the cluster completely.

Error Handling

Spark errors are difficult to debug if you do not understand the internals. Here is an excellent post on unintuitive errors that the Spark application can throw.

Code performance, cluster configuration, and error handling are three aspects of Leaky Abstractions in Spark.

Subscribe now

Comparison with other SQL-based data systems

Now that we have reviewed the challenges with Spark, let’s contrast them with other SQL-based systems. SQL-based databases have been used for a long time. E.g., PostgreSQL, Oracle DB & Snowflake. Note that Spark SQL is a processing engine and SQL databases are both processing and storage engines.

SQL databases offer better abstractions for optimal performance because they are less flexible. E.g., In SQL databases, users do not have a choice on the physical layout of data, and the database engine is tied to the catalog. In contrast, Spark can read data from file formats like CSV, JSON, Parquet, etc., and supports any hive-based metastore (catalog).

Less flexibility -> Less configurations -> Better Abstractions

Examples of these abstractions are indexes and partitions for structuring the data optimally based on the expected query patterns.

SQL databases do have some similar challenges to Spark SQL. Different ways of writing queries can lead to different query plans resulting in different peformance. Example

There are no perfect abstractions.

Support my work by subscribing. Thanks.

Resources

• RDD Research Paper: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf

• Spark errors: https://medium.com/@yhoso/resolving-weird-spark-errors-f34324943e1c

• https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/

• What Are Abstractions in Software Engineering with Examples? https://thevaluable.dev/abstraction-type-software-example/

I found this quote when listening to Patt Helland on a podcast. The quote captures the eternal tradeoff in distributed systems. Commonly understood through CAP theorem. The core idea is that the data in the system can be consistent or have high availability, but not both. Its about knowing what is more valuable, the fact that you get the data that might be stale but available right now or the data that has all the updates but is available late.

One example of this tradeoff is reflected in AWS DynamoDB read consistency modes. DynamoDB is a scalable key-value store. They have different cost structure for eventually consistent reads and strongly consistency reads. Eventually consistent reads might not have the most recent updates, however they are fast. Strongly consistent reads will have the latest updates, but the latency & the cost might be higher. Besides, there are some other disadvantages of strongly consistent reads.

This idea can be extended from databases to software quality and requirements clarification. It is always important to understand as a developer given the constraints, what is more important to business and stakeholders? Building the software with the “right”quality control mechanisms like unit testing, telemetry etc. or delivering it fast i.e. “right now”.

References

• https://www.infoq.com/news/2019/12/data-storage-trends-helland/

• https://www.se-radio.net/2020/02/episode-397-pat-helland-on-data-management-with-microservices/

Setting Expectations

Navigating a new team can be challenging, especially if you're not sure what's expected of you. There are short-term expectations like becoming familiar with the team's work and completing administrative tasks like setting up your machine and sharing your compensation preferences. Usually, there's a defined process for this.

Medium-term expectations may not be as well-defined. It's important to understand what deliverables are expected of you within the next few months and how they can help you interact with the code base. Some teams don't have established processes for new engineers, so it's crucial to ask for a plan and identify tasks that can set you up for success.

Leverage the Past

Learning from past mistakes and successes can help you propose improvements or start conversations. As a new team member, you have a unique perspective that can contribute to the team's success. For example, you may have experienced a code formatting template in a previous team that could be useful in your new team.

When proposing changes, be mindful not to come across as a know-it-all or overly negative. Since you don't have institutional knowledge, it's best to ask for the team's opinion on a certain change, which can lead to more collaborative discussions.

Cultural Differences

Different teams or organizations operate in various ways, and it's essential to understand and adapt to cultural differences. For instance, some teams prefer unstructured discussions, while others prefer more formalized meetings. It's best to align yourself with the team's culture before starting and make small changes to feel included and connect better with the team.

Better Questions

As a new team member, there are no bad questions, but some questions are better than others. Open-ended questions can prompt relevant discussions and help you understand things you might not have otherwise thought of. However, be mindful of your colleagues' time and research beforehand to avoid taking up too much of their time.

New Habits

Starting a new job is an excellent opportunity to develop new habits that can benefit you in the long run. You can establish better work-life balance or watch more tech talks, for example. The fresh start effect can help in developing better habits, and it's essential to take advantage of this opportunity.

I hope this article has provided you with some helpful tips to succeed in your new team. Remember that everyone's experience is different, and it's best to adapt to your new team's culture while staying true to yourself.

However, there are use-cases which require using libraries from a different language than the one the application was written in. One such scenario is that a Spark application is written in Scala, but now there is a need for Python libraries. This is common as Python has a better ecosystem of libraries around statistics & machine learning. E.g., SciPy, NumPy etc.

How can we leverage Python libraries in a Scala Spark application?

Spark does have a Python API (PySpark), but its core is written in Scala. Let us think about how Spark achieves that.

Below is the high-level data flow between Python runtime and JVM (Scala runtime)

Spark leverages a library called Py4J, that allows invoking of JVM code from Python runtime. This communication happens over a socket in the driver.

We can re-use the same Py4J bridge to callback Python code from within the Scala code. Below is the data flow.

When spark-submit happens, it enters the Python driver code, which just invokes the Scala Driver code of your existing application. In your Scala application you would define an interface that can be defined inside the Python runtime. In this interface implementation in Python you can leverage whatever libraries you wish to use.

Pseudocode: Python Driver

from python_implementation import PythonImplementation 
from pyspark.java_gateway import ensure_callback_server_started  

sc = //Spark context 

// This enables Java to call Python code 
ensure_callback_server_started(sc._gateway) 

python_implementation = PythonImplementation(sc._gateway, spark._wrapped) 

scalaDriver = new jvm.your.scala.Driver(python_implementation) 

scalaDriver.execute(spark._jsparkSession)

Pseudocode: Scala Interface Definition that Python Implements

trait PythonImplementation {
 def process(df: DataFrame): DataFrame 
}

Pseudocode: Scala Driver

def execute(sparkSession: SparkSession) = {
  scDf = //some dataframe	
  pythonEvaluation.evaluate(scDf) 
}

Pseudocode: Python Implementation

class PythonImplementation(object):
   def __init__(self, gateway, sql_context): 
       self.gateway = gateway 
       self.sql_context = sql_context 

   def process(self, df): 
       // Your Python logic  

   class Java: 
       implements = ['your.PythonImplementation']

Caveats

• The solution relies on Spark private variables like _jvm, _gateway etc., which may break the solution in future versions of Spark

• This does not and is not intended to solve performance problems with Python UDFs

Follow me on LinkedIn

Let’s take an example where we need to count number of managers, engineers and sales professionals from an employee dataset.

We could do something like below.

employees_df = spark.sql("select * from employees")
num_managers = employees_df.filter("employee_type is manager").count()
num_engg = employees_df.filter("employee_type is engineer").count()
num_sales = employees_df.filter("employee_type is sales_professional").count()

However, these would lead to 3 separate SQL queries being executed and your data would essentially be scanned thrice i.e. to count for each employee type.

There has to be a better way!

So.. Spark can be lazy. Spark has a good reason to be, unlike me😀

The lowest level of abstraction that Apache Spark provides are RDDs (Resilient Distributed Dataset). SparkSQL code is ultimately converted to RDDs.

RDDs support two types of operations

• Transformations: They are operation that transform an RDD. In the example that is the RDD code for the SQL query.

• Actions: Operations that trigger the transformations to compute a statistic like count/sum or show the data. In the example that is the count() call.

Spark does not process any transformations till it sees an action because its lazy. This makes sense because by the time the action is called, Spark can know all the things you wish to compute and optimize the execution of all the transformations. The later you call an action the better (i.e. the later you count your chickens the better)

The above example can now be written as below to reduce the number of actions to 1.

spark.sql("select count_if(employee_type = manager) as mgr_count, 
count_if(employee_type = engineer) as engg_count,
count_if(employee_type = sales_professional) as sales_count
from employees").first()

With the above code the only action that got executed was the call to first(). This query gives enough information to Spark to just do one pass over your data and calculate all of the counts you need.

We would have to manage less code, but we did not have the time.

In addition, when there are no safeguards like unit/integ tests, the need to just add stuff worsens the situation. This culture results in Tech Debt, which incurs the compound interest of poor readability and in some cases performance.

It is misleading to say that since customer is getting immediate benefits, delivering the code by any means is justified. Developer productivity has direct impact on the rate of the delivery over a period. It is okay to make such tradeoffs when the organization/company is starting out and the initial speed is non-negotiable. However, as organizations mature, these factors can be the difference between long term success and failure.

So the next time someone refactors or removes the code. Thank them!

References

1. https://www.bloomberg.com/opinion/articles/2021-04-24/research-shows-why-simplifying-is-hard-and-complicating-is-easy

2. https://quoteinvestigator.com/2012/04/28/shorter-letter/

Software quality is one aspect that is difficult to measure. There are metrics like test coverage, cyclomatic complexity & number of operational incidents etc. to measure software quality. Although these metrics are necessary but they are definitely not sufficient.

Readability of variable & method names is one area of quality that is difficult to measure. Besides, modularity of software can technically be measured by cohesion and coupling metrics, but there is a risk of false positives. These aspects of quality have an element of subjectivity or dependence on specific business needs.

Having expressed the difficulty in measuring these aspects, I believe it is still possible to manage these by having right guidelines and code review processes. More importantly these can be managed by developing the culture of clean code that is imbibed in every member of the team. The solution has more to do with the judgment and common sense rather than identifying complex proxies for measuring readability and modularity.

References

• https://deming.org/myth-if-you-cant-measure-it-you-cant-manage-it/

• https://martinfowler.com/bliki/CannotMeasureProductivity.html

• https://hbr.org/2010/10/what-cant-be-measured

If you found this post informative, kindly express your support by subscribing to this newsletter below.

Subscribe now

I have realized that quotes are the easiest way to retain information. “Righty tighty, lefty loosey“ is a great example. However, expressions of these ideas in a concise manner is not as easy. For appreciation and for fairness, I would try to provide reference and credits for all the quotes that I mention here.

Hope you would encourage me by subscribing below and spreading the word.

Subscribe now