When I joined PayPay, one of the first things I tried to understand was how our services were containerized.

A common exercise when joining a new backend team is to look at the Dockerfiles used across services as they often reveal interesting patterns:

• Which base images are used

• Whether the builds are optimized

• If multi-stage builds are implemented

• Opportunities for reducing image size or improving caching

So naturally, I went looking for the Dockerfile.

But to my surprise… I couldn't find one.

After digging a bit deeper, I realized something interesting. The project wasn’t using a Dockerfile at all. Instead, it was using Jib.

📦 Enter Jib

Imagine shipping a Java microservice where changing a single line of code doesn’t force your CI pipeline to rebuild a massive container image.

That’s exactly what Jib enables.

Jib is an open-source container image builder for Java applications that:

• Eliminates the need for a Dockerfile

• Does not require a Docker daemon

• Integrates directly with Maven and Gradle

Instead of packaging your application as a single fat layer, Jib understands Java project structure and splits your application into optimized layers:

• Runtime (base image)

• Dependencies

• Resources

• Application classes

Because of this layered structure, when you change only your application code, only the top layer is rebuilt and pushed.

The result?

• ⚡ Faster builds

• 🧱 Better caching

• 🚀 Simpler CI pipelines

• 🔧 Less Dockerfile maintenance

In this post, we'll explore how Jib works, when to use it, and how it compares to traditional Dockerfiles.

🚀 What's Jib?

If you build Java applications and ship containers, you’ve probably written Dockerfiles like this:

FROM eclipse-temurin:17-jre
WORKDIR /app
COPY target/app.jar app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]

It works.

But it’s not optimized for Java.

This is where Jib changes the game.

🧱 The Problem With Traditional Docker Builds

Before Jib, containerizing Java apps required:

• Writing and maintaining Dockerfiles

• Installing Docker everywhere (including CI)

• Packaging fat JARs

• Rebuilding full images for small code changes

• Manually optimizing layers

Even a small code change could invalidate caching and rebuild everything.

That’s inefficient.

🧠 How Jib Works

Jib builds container images by splitting your app into logical layers:

1. Base Image (JRE)

2. Dependencies

3. Resources

4. Application classes

This means:

• If dependencies don’t change → layer reused

• If only code changes → only top layer rebuilt

This drastically improves CI speed.

🛠️ Getting Started with Jib

Maven Setup

Add this to your pom.xml:

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.4.0</version>
  <configuration>
    <to>
      <image>ghcr.io/yourorg/yourapp</image>
    </to>
  </configuration>
</plugin>

Build & push:

mvn compile jib:build

Gradle Setup

plugins {
  id("com.google.cloud.tools.jib") version "3.4.0"
}

jib {
  to {
    image = "ghcr.io/yourorg/yourapp"
  }
}

Build:

./gradlew jib

⚔️ Dockerfile vs Jib - Which Is Better?

This is the real question.

The answer depends on some of the use cases.

✅ When Jib Is Better

1. Standard Java Microservices

• Spring Boot

• Micronaut

• Quarkus

• Plain Maven/Gradle apps

Jib integrates directly with your build system.

2. Faster CI/CD Pipelines

Jib:

• Does not require a Docker daemon

• Builds incremental layers

• Pushes smaller diffs

• Produces reproducible builds

If CI time matters, Jib usually wins.

3. Developer Simplicity

No Dockerfile. No manual layering. Less DevOps friction.

🏗️ When Dockerfile Is Better

1. OS-Level Customization

If you need:

• apt-get install

• Native libraries

• Custom shell scripts

Jib does not support arbitrary RUN commands.

Dockerfile wins here.

2. Polyglot Applications

If your container includes:

• Node + Java

• Python + Java

• Multiple build tools

• Custom entrypoints

Dockerfile provides full flexibility.

3. Complex Multi-Stage Builds

Example:

• Compile native binaries

• Build frontend assets

• Run security scans during build

• Copy artifacts between stages

Dockerfile is more powerful here.

📊 Side-by-Side Comparison

🎯 Real-World Scenarios

Scenario A: 20 Spring Boot Microservices

• Kubernetes

• GitHub Actions

• CI cost matters

👉 Jib is usually the better choice.

Scenario B: Enterprise App With Native Dependencies

• Custom Alpine base image

• OS packages

• Hardening scripts

• Multi-stage builds

👉 Dockerfile is better.

Scenario C: Small Startup Team

• Java-only stack

• No DevOps team

• Want fast pipelines

👉 Jib simplifies everything.

🔬 Performance Insight

Because Jib separates dependencies and classes:

• Code-only changes rebuild in seconds

• Docker layer cache invalidation is minimized

• CI builds are more predictable

In large microservice ecosystems, this adds up significantly.

🚀 Closing Thoughts

Jib is optimized for:

Java developer productivity.

Dockerfile is optimized for:

Infrastructure control.

If you build modern Java microservices → Start with Jib.

If you need deep OS-level control → Use Dockerfile.

Containerization doesn’t have to be complicated. If your stack is Java-centric and you value speed, reproducibility, and developer autonomy, then Jib is one of the smart tools you can adopt today.

At my current org, I got introduced to a very fun tool, PagerDuty - because who doesn't love being woken up at 3:31 AM?

A similar alert flared up for one of our merchant services. While it mercifully avoided the 3 AM slot, the diagnostic was clear: The containers running were restarting. There’s no bad deployment, just a burst of events on a Kafka topic and a trail of exit code 137.

This is the story of how we tracked down the memory issue in one of our core merchant services at PayPay that wasn't exactly a leak, but a partition mismatch, and a battle between Garbage Collectors.

The Incident: When Bursts Turn Into Crashes

It started with a Kafka topic.

This topic powers merchant payouts in Japan and, on average, processes around 3,500 messages per day. On paper, that volume sounds harmless. In reality, the traffic pattern was anything but smooth. Instead of being evenly distributed, most of those messages arrived in short, aggressive bursts.

And that’s when things began to fall apart.

Every time a burst hit, our merchant service pods would suddenly spike in memory usage. Within minutes, they would hiy their memory limit and get terminated by Kubernetes with an OOMKill.

The Investigation

When we looked at the consumer topology, the first red flag was almost too obvious to ignore:

• Kafka partitions: 6

• Active consumer pods: 10

In Kafka, this setup has a very specific behavior. A single partition can be consumed by only one consumer within a consumer group. So with 6 partitions and 10 pods, 4 pods were effectively idle, waiting on the sidelines.

Every burst of messages was being funneled into just 6 pods, concentrating the entire load on 60% of our fleet, while the remaining pods contributed nothing. Under normal traffic, this imbalance went unnoticed. But during bursts, those active pods suddenly did significantly more work than they were sized for.

That explained where the pressure was landing, but not why it was causing hard crashes.

To answer that, we moved to the next step: capturing a heap & thread dump in our staging environment while simulating a 5X production load. And that’s where things started getting really interesting.

The ZGC Struggle

We were initially running the service on ZGC.

On paper, ZGC is impressive - microsecond-level pause times and near-zero stop-the-world events. But what often gets missed is its biggest trade-off: ZGC needs a lot of breathing room. It relies heavily on having heap memoryto keep allocation and reclamation in balance.

When we profiled the system, the numbers told a worrying story. The old-generation worker alone had consumed over 584 seconds of CPU time. ZGC wasn’t idle - it was working relentlessly, trying to keep up with a flood of newly allocated objects.

But it simply couldn’t reclaim memory fast enough.

The heap kept expanding, allocation pressure kept rising, and eventually the pod crossed its memory limit and was OOMKilled. Low pause times didn’t matter if the process couldn’t stay alive long enough to benefit from them.

The Substitute: G1GC

At this point, we decided to switch strategies and flipped the JVM to G1GC, then reran the same load test.

The difference was immediate:

• Memory usage: Stabilized under 1 GiB (previously went up all the way to 3 GiB)

• Pause times: Increased slightly to around 35 ms, still comfortably within our SLAs

• Stability: No crashes. No restarts.

G1GC traded a bit of pause time for predictable memory behavior, which turned out to be exactly what this workload needed. In a burst-heavy system, consistency beats theoretical best-case latency every single time.

The Side-Quest: The Slack Notifier "Leak"

While digging through the thread dumps, we found a "Red Herring." Our SlackNotifier class, which is responsible for sending failure alerts, was instantiating a new OkHttpClient for every single error notification.

Each OkHttpClient spins up its own connection pool and thread pool. In a high-error scenario, this could have quietly expanded into dozens, if not hundreds, of threads, steadily pushing the JVM toward thread exhaustion and memory pressure.

Interestingly, this wasn’t the direct cause of the production crashes. During the Kafka bursts, no errors were actually being emitted, which meant this code path wasn’t even executed. But left unfixed, it was a ticking time bomb waiting for the next incident.

The fix was straightforward: refactor the notifier to use a single, shared OkHttpClient singleton, ensuring controlled resource usage and predictable behavior.

Not the root cause, but a critical cleanup uncovered at exactly the right time.

Final Fixes

To stabilize Merchant Finance, we didn’t just tweak application code—we re-aligned the infrastructure to match the workload it was actually handling.

Here’s what we changed.

1. Memory Alignment

We set memory.request to match memory.limit at 3 GiB.

This ensured the node reserved the full memory upfront, eliminating eviction pressure during bursts and preventing Kubernetes from stepping in with an OOMKill when memory usage spiked. [A reddit thread worth exploring]

2. Partition Scaling

We scaled the Kafka topic from 6 partitions to 10.

This change ensured that every merchant service pod became an active consumer, evenly distributing the workload instead of concentrating it on a subset of the fleet. Burst traffic stopped being a stress test for a few pods and became a shared responsibility across all of them.

3. The GC Switch

We officially moved this service to G1GC.

For this workload profile, we chose predictable memory behavior over ultra-low pause times. Slightly higher pauses were a small price to pay for stability under burst-heavy traffic.

4. CPU Buffering

We increased the CPU request from 100 millicores to 500 millicores.

This wasn’t about average CPU usage; it was about headroom. During peak allocation and garbage collection cycles, G1GC needs consistent CPU availability to do its job efficiently. An under-provisioned CPU would only delay GC work and amplify memory pressure.

5. Fixing SlackNotifier

Finally, we refactored SlackNotifier to use a single shared OkHttpClient instance instead of creating a new one per notification.

It wasn’t the root cause, but it removed a hidden risk that could have turned a future incident into something far worse.

Results

After applying the infrastructure and JVM changes, we reran the same burst-heavy load tests and monitored JVM, Kubernetes, and Kafka metrics.

The difference wasn’t subtle 👇

1. JVM Behavior (ZGC v.s G1GC)

• GC pause frequency dropped significantly

• GC avg pause times increase

The change is clearly visible around 12:00–12:10, which is when the GC and infrastructure updates were applied.

2. G1GC Under Load

Switching away from ZGC did increase pause times, but well within acceptable limits.

• Average GC pause: ~30–40 ms

• Consistency: Flat, predictable, no spikes

• Impact: No effect on SLAs

This was an intentional trade-off. We gave up ultra-low pause times in exchange for a lower GC pause count and memory predictability.

3. Kubernetes Pods Stopped Restarting

From a platform perspective, this was the most important outcome.

• Pod restarts: 0

• OOM events: None

• Memory usage: Stable across all pods

• CPU usage: Predictable, with sufficient headroom during GC cycles

Container-level metrics after the fixes - memory usage remains stable, CPU has sufficient headroom, and pod restarts drop to zero.

4. Kafka Kept Up With Bursts Without Errors

Finally, we looked at the Kafka side of the system.

• Consumer lag: Spiked briefly during bursts, then drained smoothly

• Consumer latency (P95): Stable

• Errors: None observed

• Throughput: Fully sustained during burst windows

Increasing the partition count ensured every pod actively participated in consumption, preventing load concentration and smoothing out processing during spikes.

Kafka consumer metrics during burst traffic. Lag increases temporarily but recovers quickly, with no errors and stable latency.

Final Outcome

After all fixes were applied:

• ✅ No pod restarts during burst traffic

• ✅ No OOMKills

• ✅ Predictable JVM behavior under load

• ✅ Kafka bursts handled without errors

Resilience isn’t just about handling errors after they happen, it’s about understanding how your system behaves under load.

If you’re running JVM workloads on Kubernetes, it’s worth taking a hard look at both your garbage collector choice and your resource requests versus limits. These decisions don’t show their impact during steady state traffic, but they matter enormously when load arrives in bursts.

Sometimes, the “latest and greatest” option isn’t the right fit for every workload. Stability often comes from choosing the tool that behaves predictably under stress, not the one that looks best on paper.

In production, boring and predictable almost always win.

At my previous organization, we were building a health platform, and one of the most critical, high-traffic components was our Lab test booking service. One of its core functions is to serve package details like inclusions, prices, and provider information for thousands of user requests daily.

The initial architecture was simple: every single request meant a new query to a database. As traffic scaled, our database struggled. Latency spiked, and CPU usage became a constant concern.

With mostly static data that rarely changes, the clear solution was to cache it. But the execution is where we got creative. This is the story of how we implemented a zero-cost, in-memory cache in our Node.js microservices and used a Kubernetes feature, “The Headless Service” to handle distributed cache invalidation flawlessly.

🛑 The Problem: Fast Caching & Scalable Invalidation

We decided on in-memory caching using a library like @nestjs/cache-manager to slash DB load and boost response times. However, in a Kubernetes cluster, where our service runs across multiple, dynamic pods, this creates a major headache: Cache Inconsistency.

If an admin updates a package price in the MySQL database, only the single pod that handled the write sees the change immediately. The other pods are serving users stale data - consistency, gone.

Furthermore, Kubernetes pod IPs are temporary. They change during restarts, deployments, and scaling events. We couldn't rely on hardcoded lists. Our invalidation system needed a reliable way to discover every running pod at the exact moment of a data update.

We needed a broadcasting mechanism.

🛠️ The Solution: The K8s Headless Service

Our strategy was a hybrid approach: local caching for performance, and a targeted internal HTTP broadcast for invalidation.

Step 1: Headless Services for Pod Discovery

This is the key to the whole solution. A standard Kubernetes service acts as a load balancer with a single Virtual IP. But a Headless Service is different. By setting clusterIP: None in the Service manifest (.spec.clusterIP), Kubernetes skips the load balancer and, crucially, returns a list of individual DNS A records for every active pod’s IP address.

Here is the simple(yet powerful) Headless Service manifest we deployed:

apiVersion: v1
kind: Service
metadata:
  name: booking-service-headless
spec:
  clusterIP: None             # THE TRICK: Makes it Headless
  selector:
    app: booking-service
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000

Now, when we resolve booking-service-headless.default.svc.cluster.local, we get an array of all current, healthy pod IPs.

For visualization, here's a diagram of Headless Services in action:

Step 2 & 3: Broadcast Implementation

In the next step, we added kubectl into our Docker image for API queries. This gives the pod the power to query the Kubernetes API directly.

To implement this safely, we followed the Principle of Least Privilege. We created a dedicated ServiceAccount for our booking service and bound it to a specific Role using RBAC (Role-Based Access Control). This Role was strictly limited: it only allowed the list and get operations on the endpoints resource within its own namespace. This ensures that even if a pod is compromised, an attacker cannot delete resources or view sensitive secrets.

Second, we created a simple, internal /invalidate-cache API endpoint on our booking service. When we hit this endpoint, it clears a specific cache key (passed in the request body) using the in-memory cache’s del() method.

When an admin update happens, and the data is written to MySQL, the pod that handles the write executes a bash script using kubectl and curl:

It uses kubectl to get the list of IPs from the Headless Service endpoints, then sends a direct HTTP POST request to the internal /invalidate-cache endpoint on each pod.

# Executed by the Node.js process on update
pod_ips=$(kubectl get endpoints listing-headless-service -o jsonpath='{.subsets[0].addresses[*].ip}')

for ip in $pod_ips; do
   curl -X POST http://$ip:3000/v2/invalidate\
       -H "Content-Type: application/json" \
       -d '{"key": "package-123"}' # Clears the specific key
done

This ensures the system is:

• Dynamic: The IP list is fresh at invalidation time, handling upscaling, downscaling and restarts seamlessly.

• Targeted: We clear only the stale package key, maximizing cache retention.

• Cost-Effective: Zero new service bills.

• Reliability: Add curl retries and logging for failed broadcasts.

For those looking to keep their Docker images clean, you can avoid installing kubectl and curl entirely by using the official @kubernetes/client-node library. By using the library, your Node.js process can query the Kubernetes API directly from within the code. This makes the logic more testable, allows for better error handling, and removes the overhead of spawning shell processes. It turns your infrastructure logic into standard application code.

📈 Production Results and Takeaways

Post-rollout, our MySQL database load has consistently remained low, and our cache hit rate has remained above 95%+. We have had zero reported incidents of users seeing stale package data. The Headless Service trick proved robust, even during high-traffic deployments.

This solution proves that sometimes, the most elegant and cheapest solution isn't an expensive managed service, but a creative application of your existing tools. It’s about being smart with your infrastructure, not just throwing money at the problem.

At our current organization, earlier this year as we were looking at the errors at one of our signup API, we observed that nearly 5% of our requests were getting failed, all due to 400 BAD REQUEST errors, and the root cause was traced back to a regex check.

This regex was a constraint that our system allows only English characters for users to input their first and last names, yet many individuals opted to enter their names in their native languages. Particularly, these customers were the people who were interested in purchasing health policies from our platform hence making them a crucial segment of our user base.

In response to this, we decided to address the 5% case by enabling users to enter their names in any language they preferred. However, this brings up a lot of challenges that we need to solve.

Challenges

1. Data Storage Strategy

We rely on MongoDB for storage and retrieval of user names. While MongoDB allows storage for all UTF-8 compatible characters, the problem comes when dealing with search. For English names, our search operations utilize the simple collation method. The corresponding fields are appropriately indexed to optimize query performance.

While the option to implement a collation index for other languages also exists in MongoDB, this approach necessitates informing DB about the specific language for which we intend to search. The challenge here is that our user base spans many languages, with India alone having more than 20 diverse languages.

Since our objective was to extend support to at least all Indian languages, implementing collation indexes for every supported language would lead to an increased number of indexes and hence an increase in index size over time as well.

Additionally, this approach would place the responsibility on developers to remember to add an index for each new language as our language support expands, which is far from an efficient solution.

2. API Gateway Constraint

All our APIs are exposed behind an API gateway and just before the gateway forwards a request to the respective API service, an inbound policy verifies the user's authentication status. Once the user is authenticated, it retrieves basic user details such as name, mobile number, and other metadata, and appends it to a request header of that API.

A multitude of APIs rely on this user-specific data in headers for their further processing.

However, there's a restriction imposed by the gateway – it allows only ASCII characters for processing and inclusion in the headers. So we had to make sure even though the name is in any other language, the response we share must be exclusively in English.

Also, this process must remain fast, as any delay in authentication could lead to sluggish API performance.

3. External Partners' Challenge with Vernacular Names

Even if we started to accept names in multiple languages, there were our partners who had to accept those names from us. If they don't support multilingual names, the user journey breaks. One such was our payment partner. We had to make sure our payments team always received the English name even when users provided names in other languages.

Also, we wanted to avoid those annoying pop-ups prompting users to enter their names in English whenever we needed to. So keeping these problems in mind we had to build a viable solution.

Solutioning

While utilizing a third-party transliteration service might have been the easiest route, we opted to develop an in-house solution to control costs and maintain full control.

Considering the API gateway and the requirements of payment partners, it became clear that we needed to transform non-English names into English equivalents. However, presenting this English name to the user was counterintuitive – for example entering a name in Hindi, only to see it transformed into English upon logging in seemed contradictory.

Therefore, we developed a dual-naming strategy. The original fields, "firstName" and "lastName" would retain the user-entered names in their entered language, while we introduced two additional fields, "englishFirstName" and "englishLastName" dedicated to storing the English counterparts of these names. These English names could then be shared with the API gateway and our payment partners.

Coming back to the challenge of storing these names efficiently, we anticipated that managing collation indexes as the number of supported languages grew would become unmanageable. Additionally, searching would require specifying the collation for each query, creating an added layer of complexity. Hence, we decided to pivot away from this approach.

Our second approach involved using Unicode. As we aimed to support multiple languages without constraint, we recognized that Unicode could effectively represent characters in nearly every language. Consequently, we decided to store Unicode representations for first and last names in their respective MongoDB fields.

We just added another layer between our DB and the application that converts these Unicode strings to the original values in the local language while retrieving the names from DB and converting the local names to their respective English names for storing them in englishFirstName and englishLastName at the time of any insert or update.

This strategy provided the flexibility we needed to manage multilingual names seamlessly.

Key Design Considerations

1. Unicode Optimization

Unicode representation typically comprises a 6 character string, with 'a' represented as 'U+0061' and 'P' as 'U+0050,' commonly commencing with 'U+00.' To conserve space in our database storage, we opted to omit the 'U+' prefix and leading zeros, optimizing our data storage.

2. Transliteration vs. Translation

Initially, our aim was transliteration, which requires converting names from one script to another while retaining their phonetic sound. For example, the Hindi word "प्रतीक्षा" should be transformed to "Partiksha" and not translated to its English equivalent i.e. "Wait". However, we recognized that Google Translate primarily focuses on translation, not transliteration. Again we didn't want to go directly for the paid Google transliterate service in our first iteration hence we developed our transliteration service using the free version of Google translate.

3. Contextual Enhancements

Another and the most crucial observation that we had was providing context to the Google Translate API that influenced its responses. To leverage this, we experimented with adding statement prefixes to non-English names to establish context. After a few hits and trials, we realized that for shorter names (less than 5 characters), a more extensive prefix statement didn't yield desirable results, and Google often returned the same Hindi word. For longer names, we employed lengthier statements, determining the optimal balance through trial and error.

Translating names normally led to their literal translation. For example "प्रतीक्षा" to "Wait" instead of "Pratiksha":

Adding a prefix statement corrected it:

Initial Code

After our first iteration, we developed the below code for transliteration. Here we are using the @iamtraction/google-translate library which is a wrapper written over free Google translate API.

const translate = require('@iamtraction/google-translate');

function getGoogleTranslateText(localName) {
  /*
    Adding an English sentence before the name so that
    it doesn't get translated to its literal meaning.
    For eg परीक्षा to Exam instead of Pariksha.
  */
  if (localName.length <= 5) {
    return `name: ${localName}`;
  }
  return `your name is: ${localName}`;
}

async function translateNameToEnglish(localName) {
  if (localName.match(/^[a-zA-Z ]+$/i)) {
    // If the name is already in English just return
    return localName;
  }
  try {
    const res = await translate(getGoogleTranslateText(localName), {
      to: 'en',
    });
    const translatedName = res.text.split(':')[1].trim();
    return translatedName;
  } catch (err) {}
  // In case of error, Return the Unicode string
  return localName;
}

Beta Release and Production Challenges

Once we built this, we released the feature in beta, and approximately 250 users signed up with non-English names within the first few days.

Upon simple eyeballing on translated texts, we found out that the process of converting the name from its local language to Unicode was working perfectly fine and users were able to view their names properly in the application in the language they preferred but we identified two issues as far as the process of transliteration to English was concerned:

1. Some names were incorrectly transliterated. This occurrence could be attributed to our dependence on Google Translate, a general translation service, rather than a specialized transliteration service.

2. Some names remained unaltered and were not transliterated. These names were returned in the same language as the original one. This meant adding context with prefix sentences before the translation was causing problems for specific names.

This prompted a further investigation that led us to another npm package called "unidecode," which converts Unicode to the original string. While initial tests with unidecode showed accuracy, they also revealed minor spelling discrepancies. In contrast, Google consistently delivered translations with correct spellings. We just needed to find a way of using the best of both worlds.

Hence, we incorporated unidecode into our algorithm as part of our solution.

Improved Solution

const translate = require('@iamtraction/google-translate');
const unidecode = require('unidecode');
const { isAlmostEqualStrings } = require('./levenshtein');
/**
 *
 * @param {String} localName
 * @description Generates text for Google (shorter statement context for short names) based on localName length
 * @returns {String} returns text to translate
 */
function getGoogleTranslateText(localName) {
  /*
    Adding an English sentence before name so that
    it doesn't get translated to its literal meaning.
    For eg परीक्षा to Exam instead of Pariksha.
  */
  if (localName.length <= 5) {
    return `name: ${localName}`;
  }
  return `your name is: ${localName}`;
}

/**
 *
 * @param {String} localName
 * @description Give an ALMOST transliterated name
 * @returns {String} returns a converted transliterated name from the local language
 */
function transliterate(localName, googleTranslatedName) {
  const decodedName = unidecode(localName);
  if (
    decodedName &&
    Array.from(decodedName)[0]?.toLowerCase() !==
      Array.from(googleTranslatedName)[0]?.toLowerCase() &&
    !isAlmostEqualStrings(decodedName, googleTranslatedName)
  ) {
    return decodedName;
  }
  return googleTranslatedName;
}

/**
 *
 * @param {String} Input non English string
 * @description translates non-english string to English
 * @returns {String} returns translated string
 */
async function translateNameToEnglish(localName) {
  if (!localName || localName.match(/^[a-zA-Z ]+$/i)) {
    // If name is already in English just return
    return localName;
  }
  try {
    const res = await translate(getGoogleTranslateText(localName), {
      to: 'en',
    });
    const translatedName = res.text.split(':')[1].trim();
    return transliterate(localName, translatedName);
  } catch (err) {}
  // In case of error, Return original string
  return localName;
}

After obtaining the translated name, we feed it into the recently introduced transliterate function. Inside this function, our initial step involves extracting the decoded string using the Unidecode library. However, the crux of the matter arises: How do we determine which result to prioritize, i.e., the decoded string or the translated string?

To tackle this, we implemented Levenshtein Distance, an algorithm that calculates the similarity between two strings.

Initially, we check if the first character of the decoded name matches the first character of the translated name. If it's not, then for sure the translated name was incorrect hence we return the decoded name, even though it might contain minor spelling discrepancies it's better to use that than the incorrect translation.

If it does match then we go for the Levenshtein Distance algorithm.

The Levenshtein distance is a number that tells you how similar two strings are. The higher the number, the more dissimilar the two strings are

In the implementation, we have a function isAlmostEqualStrings that generates a value from 0 to 1 and returns true if the value is above a certain threshold. In our case, we set the threshold to 0.8

If the Levenshtein distance indicates a match exceeding 80%, we return the translated name; otherwise, we return the decoded name. This approach ensures that we prioritize accuracy, offering a reliable result based on the established similarity threshold.

This updated algorithm substantially reduced the above mentioned issues. Even though it's not 100% accurate it solved our 5% cases very well.

Conclusion

The algorithm developed was entirely in-house and incurred no costs. While investing in a paid solution might have potentially offered better results, wise engineering decisions taken iteratively and a handful of clever hacks played a vital role in both reducing costs and efficiently resolving the specific problem we had.

The complete code for the above implementation along with the Levenshtein Distance algorithm can be found on GitHub (contributions/corrections are welcome).

With this, we come to the end of the article. My DMs are always open if you want to discuss further on any tech topic or if you've got any questions, suggestions, or feedback in general:

• Twitter
• LinkedIn

Happy learning!

As a developer, how often have you seen a repository with packages that are out of date?

New package updates generally include new features, performance improvements, and security fixes. But keeping track of all outdated dependencies in your project can be really boring and a time consuming task, especially if you have lots of them.

So to do this sort of housekeeping, I tried out Dependabot.

How Dependabot Works

Dependabot goes through the dependency files of your project. For instance, it searches your package.json or pom.xml files and inspects for any outdated or insecure dependencies. If it finds any, it opens individual pull requests to update each one of them.

This tool is natively integrated with GitHub. But recently, I had to solve this problem of updating dependencies for a project running in Azure DevOps. So I decided to find a workaround to integrate Dependabot with Azure Pipelines. In this blog post, I will share my solution.

If you go to Azure DevOps Extension Marketplace and search for "Dependabot", you will find an extension by Tingle Software. Using this extension we can easily integrate Dependabot with our repos in Azure DevOps.

You can check if you have this extension in your "Organization Settings" in Azure DevOps. If not, make sure you have it installed before proceeding.

Installed Extensions - Azure DevOps

How to Create the Azure Pipeline

Let's now start with creating a new YAML file for your Azure pipeline:

trigger: none

stages:
  - stage: CheckDependencies
    displayName: 'Check Dependencies'
    jobs:
      - job: Dependabot
        displayName: 'Run Dependabot'
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: dependabot@1
            displayName: 'Run Dependabot'
            inputs:
              packageManager: 'npm'
              targetBranch: 'develop'
              openPullRequestsLimit: 10

In the task parameters, I have specified three params:

1. packageManager: It specifies the type of packages to check for dependency upgrades. Examples: nuget, maven, gradle, npm, and so on.

2. targetBranch: It is an optional parameter that defines the branch to be targeted when creating pull requests. When not specified, Dependabot will pick the default branch of the repository.

3. openPullRequestsLimit: This is again an optional parameter that specifies the maximum number of open pull requests to have at any one time. By default, it opens 5 pull requests at a time.

You can go through all the Task Parameters that the extension supports to tweak your implementation. Now just simply configure this YAML file with a new azure pipeline and then you're ready to run it.

Pipeline Configuration - Azure DevOps

The next step is to give your repository's Project Collection Build Service access so that Dependabot can create the pull request to your project's repositories.

For that, go to your project settings. Here, you click on the repositories and search for the repo where you have integrated the pipeline.

After you have selected that, click on the security tab and search for project collection build service. You have to allow the following access to it:

• Contribute

• Contribute to pull request

• Create Branch

• Create Tag

• Force Push

Access to raise PR in Repo

With this, you're completely ready to run the pipeline. Once you do it, you'll start receiving pull requests in your repository with the updated packages.

PR raised by Dependabot

How to Schedule the Pipeline

Up to this point, you've had to manually trigger the pipeline to run. To make it run automatically, you can configure schedules for your pipelines. This will trigger your pipeline to start based on a schedule.

Use the following syntax and add it to the very top of your YAML file:

schedules:
- cron: string
  displayName: string
  branches:
    include: [ string ]
  always: boolean

The branches’ include parameter specifies which branches the schedule applies to.

The always parameter specifies whether to "always" run the pipeline or only if there have been any source code changes since the last successful scheduled run. The default is false.

For this case, you set its value to true as Dependabot updates are independent of any code changes.

The time zone for cron schedules is UTC and cron syntax is as follows:

mm HH DD MM DW
 \  \  \  \  \__ Days of week
  \  \  \  \____ Months
   \  \  \______ Days
    \  \________ Hours
     \__________ Minutes

So if you want to run your pipeline every week on Sunday at 12pm UTC, you would need to write - cron: "0 12 * * 0" (update the cron to suit your needs).

This is how your final YAML should look like after adding a schedule:

schedules:
  - cron: "0 12 * * 0"
    displayName: Weekly Dependency Updates
    branches:
      include:
      - develop
    always: true

trigger: none

stages:
  - stage: CheckDependencies
    displayName: 'Check Dependencies'
    jobs:
      - job: Dependabot
        displayName: 'Run Dependabot'
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: dependabot@1
            displayName: 'Run Dependabot'
            inputs:
              packageManager: 'npm'
              targetBranch: 'develop'
              openPullRequestsLimit: 10

This pipeline does the following for you:

It runs every week (Sunday 12pm UTC in this case) and looks for any outdated or insecure dependency. If it finds any, it opens pull requests to update each one of them individually.

Hopefully, this will help you to keep your project dependencies up to date in Azure DevOps!

Wrapping Up

With this, we come to the end of the article. My DMs are always open if you want to discuss further on any tech topic or if you've got any questions, suggestions, or feedback in general:

• Twitter
• LinkedIn

Happy learning!

At my current org, we have many applications running behind the API gateway that interacts with the outside world using REST APIs. To track server-side failures we built an alert system that nudges every time we get any 5XX error.

As soon as we made the alert system live, we started noticing a few HTTP 502 “Bad Gateway” errors occurring every day but intermittently.

As per RFC7231, the 502 (Bad Gateway) status code indicates that the server while acting as a gateway or proxy, received an invalid response from an inbound server it accessed while attempting to fulfill the request.

With very little known information, I began chasing down the RCA for these failures.

Debugging

As I started debugging the issue I wanted first to check what was going on in the application at the time when the particular request arrived. However, I discovered that this failed request never reached the server-side application.

We also have a proxy server sitting in between our API gateway and the application so I checked the logs of that as well and fortunately, I was able to track the request there.

I observed the request despite coming from the API gateway, the response sent was 502, without even communicating with my application.

Next, I checked the throughput when such a response was sent as I thought maybe there was a limitation in the number of concurrent requests our application can handle. But the result was again surprising because even when the throughput was < 1000 TPM (Transactions Per Minute), the 502 error persisted and we had already observed our applications working well even at a TPM of well over 5000.

Based on these findings, I realized that issue is not with our application code but with how our proxy server interacts with my application. So I started diving deep into how HTTP connections are made during the request-response cycle.

Diving in TCP Connection

In the realm of web communication, HTTP (Hypertext Transfer Protocol) initiates a unique TCP (Transmission Control Protocol) connection for every request. Setting up a TCP connection requires a three-step handshake process with the server before transmitting data. Once the data transmission is finished, a four-step process is implemented to terminate the same connection.

In some scenarios where limited data is transmitted, the three and four-step processes involved in establishing and terminating TCP connections can add a significant overhead. To address this issue, rather than opening and closing a socket for each HTTP request, it is possible to keep a socket open for a more extended period and leverage it for multiple HTTP requests.

This mechanism is known as HTTP keep-alive, which permits clients to reuse existing connections for several requests. The decision of when to terminate these open TCP sockets is managed by timeout configurations on either the client or target server or both. This approach improves the efficiency of web communication and reduces latency.

An important point here to notice is that for a short amount of time, there is a possibility that a connection can be “half-open”, in which one side has terminated the connection but the other has not. The side that has terminated can no longer send any data into the connection, but the other side still can. The terminating side continues reading the data until the other side terminates as well.

Now coming back to our story, the proxy server acts as a messenger passing requests and responses back and forth. If the service returns an invalid or malformed response, rather than transmitting that meaningless information to the client, the proxy server sends a 502 error message.

This suggests that our application might have attempted to terminate the TCP connection, but the proxy server was unaware of this and continued to send the request to the same socket. This provided us with a critical hint in our investigation.

Debugging, further

The 502 Bad Gateway error could occur when the proxy server sends a request to the application, and simultaneously, the service terminates the connection by sending the FIN segment to the same socket. The proxy socket acknowledges the FIN and starts a new handshake process.

Meanwhile, the socket on the server side has just received a data request referring to the previous (now closed) connection. As it is unable of handling it, it sends a RST segment back to the proxy server. The proxy server then returns a 502 error to the user.

Based on this hypothesis, we found the default HTTP keep-alive time. While the proxy server's keep-alive time was set at 75 seconds, our application had a keep-alive of only 5 seconds. This suggests that after 5 seconds, our application could close the connection, and while this process is ongoing, the proxy server could receive a request. As it has not yet received the FIN, it sends the request down the dead connection, receives nothing in response, and therefore throws a 502 error.

To verify this hypothesis, I modified the HTTP keep-alive timeout of our application to 1 ms in our development environment. We then conducted load testing on our APIs and found a significant number of API failures due to the same 502 error. This confirmed our initial hypothesis.

Fix

A possible solution is to ensure that the upstream (application) idle connection timeout is longer than the downstream (proxy) connection timeout.

In practice, this can be achieved by increasing the keep-alive time of the application server to a value greater than the proxy's connection timeout. For example, in our setup, I have increased the server's keep-alive time from 5 seconds to 76 seconds, which is 1 second longer than the proxy server's connection timeout.

By doing so, the downstream (proxy) server becomes the one to close the connections instead of the upstream server. This ensures that there is no race condition between the two servers and eliminates the possibility of a 502 error caused by a closed connection. Therefore, setting appropriate upstream and downstream timeouts is crucial for the smooth functioning of web applications.

The below example illustrates how to modify the default keep-alive timeout for an Express app:

const express = require('express');
const app = express();
const server = app.listen(3000);

server.keepAliveTimeout = 76 * 1000 // Time in ms

Conclusion

Theoretically, this race condition can occur anytime when the connections are reused & if the upstream service drops connections before the downstream.

In my case, it was the proxy server that pre-connects to all the backend servers, and the TCP connection persists for reusing for further HTTP requests. The Keep-Alive time of this connection was 75 seconds from the proxy and 5 seconds from the app server.

This was causing a condition where the proxy thinks a connection is open, but the backend closes it. And when it sends the request down the same connection, instead of getting the TCP ACK for the sent request, the proxy gets TCP FIN (and eventually RST) from the upstream.

The proxy server then just gives up on such requests and responds immediately with HTTP 502 error to the client. And this is completely invisible on the application side! That's why no application logs were visible to us.

To mitigate this you can either make sure the upstream (application) idle connection timeout is longer than the downstream (proxy) connection timeout OR implement retries on HTTP 5xx errors from the API gateway.

I hope you found it helpful. Comments or corrections are always welcome.

Any application that communicates with other resources over a network has to be resilient to transient failures.

These failures are sometimes self-correcting. For example, a service that is processing thousands of concurrent requests can implement an algorithm to temporarily reject any further requests until its load gets reduced. An application that is trying to access this service may initially fail to connect, but if it tries again it might succeed.

While designing any system it is essential to make it resilient against such failures. In this article, we will look at one of the many ways of achieving it using Retries.

At our current organization, we use this mechanism throughout our microservices to make sure that we handle failures and at the same time, provide the best of our services to our customers.

Let’s start by first defining what we mean by failures.

What is a failure?

Failures can be caused by numerous reasons while our services communicate with each other over a network. Some examples of types of failures are:

• A slow response / No response at all

• A response in the incorrect format

• A response containing incorrect data

In planning for failures, we should seek to handle each of these errors.

Retry

Retry is a process of automatically repeating a request in case any failure is detected. This helps return fewer errors to the users, improving the consumer experience on our application.

The only caveat when it comes to retrying is that multiple requests to the same resource should have the same effect as making a single request i.e the resource should be idempotent.

In REST APIs, GET, HEAD and OPTIONS methods generally do not change the resource state on the server and hence are mostly retryable.

When should we retry a request?

Ideally, we should only retry a failed request when we know it has any possibility of succeeding the next time otherwise it will just be a waste of resources (CPU, Memory, and Time).

For example, you might get an error with status code 503 (Service unavailable) when the server had an unexpected hiccup while connecting to a DB host. A retry may work here if the second call to the upstream service gets a DB instance that is available.

On the other hand, retrying for errors with status code 401(Unauthorised) or 403 (Forbidden) may never work because they require changing the request itself.

The general idea is that if an upstream request fails, we immediately try again, and probably the second request will succeed. However, this will not always benefit us. The actual implementation should include a delay between the subsequent requests. We will discuss that in the next sections.

Problem with normal retries

Consider what happens when we get 100,000 concurrent requests & all hosts of the upstream service are down at that instance. If we were to retry immediately, these 100,000 failed requests would retry immediately on a static interval. They would also be combined with new requests from the increasing traffic and could make the service down again.

This creates a cycle that keeps repeating until all the retries are exhausted and will also eventually makes the upstream service overwhelm, and might further degrade the service that is already under distress. This is a common computer science problem also known as the Thundering Herd problem.

To tackle this, we can introduce some delay in retrying a failed request.

Backoff

The wait time between a request and its subsequent retry is called the backoff.

With backoff, the amount of time between requests increases exponentially as the number of retry requests increases.

The scenario we just discussed above is where having a backoff helps, it changes the wait time between attempts based on the number of previous failures.

From the above figure, let's say the initial request goes at the 0th millisecond and fails with a retryable status code. Assuming that we have set up the backoff time of 200ms and neglecting the request-response time, the first retry attempt happens at 200th milliseconds (1*200ms). If this fails again, the second retry then happens at 400ms (2*200). Similarly, the subsequent retry happens at 800ms till we exhaust all the retries.

If any of the request failures are caused by the upstream service being overloaded, this mechanism of spreading out our requests and retries gives us a better chance of getting a successful response.

Jitter

The Backoff strategy allows us to distribute the load sent to the upstream services. Yet, turns out it isn't always the wisest decision because all the retries are still going to be in sync which can lead to a spike on the service.

Jitter is the process of breaking this synchronization by increasing or decreasing the backoff delay to further spread out the load.

Coming back to our previous example, let’s say our upstream service is serving the maximum load that it can handle and four new clients send their requests which fail because the server could not handle that amount of concurrent requests. With only backoff implementation, let's say after 200 milliseconds we retry all 4 failed requests. Now, these retry requests would also fail again for the same reason. To avoid this, the retry implementation needs to have randomness.

Implementation

Going through this AWS documentation that goes deep into the exponential backoff algorithm, we have implemented our retry mechanism using Axios interceptor

The example is written inside of a Nodejs application, but the process will be similar regardless of which JavaScript framework you’re using.

In this, we expect the following settings:

Total Retries The number of maximum retries you want before returning a failed response to the client.

Retry Status Codes The HTTP status codes that you want to retry for. By default, we have kept it on for all status codes >=500.

Backoff This is the minimum time we have to wait while sending any subsequent retry request.

axios.interceptors.response.use(
  async (error) => {

    const statusCode = error.response.status
    const currentRetryCount = error.response.config.currentRetryCount ?? 0
    const totalRetry = error.response.config.retryCount ?? 0
    const retryStatusCodes = error.response.config.retryStatusCodes ?? []
    const backoff = error.response.config.backoff ?? 100

    if(isRetryRequired({
      statusCode, 
      retryStatusCodes, 
      currentRetryCount, 
      totalRetry})
    ){

      error.config.currentRetryCount = 
          currentRetryCount === 0 ? 1 : currentRetryCount + 1;

     // Create a new promise with exponential backoff
     const backOffWithJitterTime = getTimeout(currentRetryCount,backoff);
     const backoff = new Promise(function(resolve) {
          setTimeout(function() {
              resolve();
          }, backOffWithJitterTime);
      });

      // Return the promise in which recalls Axios to retry the request
      await backoff;
      return axios(error.config);

    }
  }
);

function isRetryRequired({
  statusCode, 
  retryStatusCodes, 
  currentRetryCount, 
  totalRetry}
 ){

  return (statusCode >= 500 || retryStatusCodes.includes(statusCode))
          && currentRetryCount < totalRetry;
}

function getTimeout(numRetries, backoff) {
  const waitTime = Math.min(backoff * (2 ** numRetries));

  // Multiply waitTime by a random number between 0 and 1.
  return Math.random() * waitTime;
}

While making an Axios request you have to make sure to add the variables in the request configurations:

const axios= require('axios');
const sendRequest= async () => {
      const requestConfig = {
             method: 'post',
             url: 'api.example.com',
             headers: { 
                'authorization': 'xxx',
              },
              data: payload,
              retryCount : 3,
              retryStatusCodes: ['408', '429'],
              backoff: 200,
              timeout: 5000
          };
      const response = await axios(requestConfig);
      return response.data;
}

When configuring the retry mechanism, it is important to tune the total retries, and maximum delay together. The goal is to tailor these values keeping in mind the worst-case response time to our consumers.

Pictorially this is how it will work:

For Java based applications, the same can be done using resilience4j

Conclusion

In this post, we looked at one of the many service reliability mechanisms: Retries. We saw how it works, how to configure it and tackle some of the common problems with retries using backoff and jitter.

I hope you found it helpful. Comments or corrections are always welcome.

GitHub Link for the above source code can be found here

Choosing which database to use is one of the most important decisions you can make when starting working on a new app or website.

If you realize down the line that you’ve made the wrong choice, migrating to another database is very costly and sometimes more complex to do with zero downtime.

Taking time to make an informed choice of database technology upfront can be a valuable early decision for your application.

(A bonus reason for why this is important is because understanding the different DBs and their properties & which one to choose over the other is quite commonly asked in job interviews)

Each database technology has advantages and disadvantages. Cloud providers like Amazon offer various database and storage options making it harder to figure out which one is the right one.

Relational databases have been a primary data storage mechanism for a long time. Non-relational databases have existed since the 1960s but recently gained traction with popular options such as MongoDB.

In this article, we will go through the factors you should consider while choosing any database.

The language determines the database

First things first. The language or the technology you are going to use never determines the database ❌

We’ve grown accustomed to technology stacks, such as MERN (MongoDB, Express, React, Node.js) & LAMP (Linux, Apache, MySQL, PHP).

There are certain reasons why these stacks evolved. But don’t assume these are the rules. You can use a MongoDB database in your Java project. You can use MySQL in your Node.js. You might not have found as many tutorials, but it's your requirements that should determine the database type & not the language.

Understanding the Tradeoff

Let's now understand the basic tradeoffs we need to deal with while making the DB decision.

The reason that we have many database options available today is due to the CAP Theorem. CAP stands for Consistency, Availability, and Partition tolerance.

Consistency means that any read request will return the most recent write.

Availability means all (non-failing) nodes are available for queries & must respond in a reasonable amount of time.

Partition Tolerance means that the system will continue to work despite node failures.

Only two of these 3 requirements can be fulfilled at any given time. If you're building a distributed app then partition tolerance is a must. So the choice remains whether we want our database to be highly available or highly consistent.

Structure of Data

If your data needs to be structured or there is a relation between different types of data that you're going to keep and at the same time you want strict data integrity then a relational database should be a better choice for you.

For example, let's say we are making a student management application where we need to store the information of certain courses and every course needs to be taken by one or more students. In this case, we can create 2 tables (Students and Courses) where the value in the Student ID column inside the Courses table points to rows in the Students table by the value of their ID column.

Apart from this if you want your DB to be ACID compliant for example in cases when you're handling payments and transactions in your application then in that case too you should prefer the SQL based databases because the counterpart i.e the NoSQL database offers weak consistency.

ACID compliance protects the integrity of your data by defining exactly what a transaction is and how it interacts with your database. It avoids database tables from becoming out-of-sync, which is super important for financial transactions. ACID compliance guarantees the validity of transactions even in failures, disastrous events, and more.

On the other hand, if your data requirements aren’t clear or if your data is unstructured, NoSQL may be your best choice.

The data you store in a NoSQL database does not need a predefined schema. This provides much more flexibility and less upfront planning when managing your database.

A NoSQL database is a much better fit to store data like article content, social media posts, sensor data, and other types of unstructured data that won’t fit neatly into a table. NoSQL databases were built with flexibility and scalability in mind and follow the BASE consistency model, which means:

Basic Availability

This means that while the database guarantees the availability of the data, the database may fail to obtain the requested data, or the data may be in a changing or inconsistent state.

Soft state

The state of the database can be changing over time.

Eventual consistency

The database will eventually become consistent, and data will propagate everywhere at some point in the future.

The structure of your data is the most important factor in deciding whether to use a SQL or NoSQL database, so put a lot of thought into this before making a decision.

Query Patterns

The next factor to consider is how you’ll query your data. This is one of the main ways to find the best database for your use case

Do you need retrieval by a single key, or by various other parameters? Do you also need a fuzzy search on the data?

• Use Non-Relational databases if you are going to fetch data by key, then all you need is a key-value store (e.g. DynamoDB).

• On the other hand, if you will require to query many different fields you can choose both Relational DB (e.g.MySQL) or Document DB (e.g.MongoDB).

• In case you are looking for fuzzy search query capabilities (free text search), then search engines like Elasticsearch(Which also comes under NoSQL DBs) are the best fit.

• But in case your data is nicely structured and organized, it is very efficient to query your data with a SQL database.

SQL is a popular query language that has been around for over 50 years now, so it’s highly mature and well-known. It efficiently executes queries and retrieves using JOINs and edits data quickly. It’s very lightweight and declarative.

Consistency

Is strong consistency required (read after write) or eventual consistency is OK?

In case you need to read your data right after your write operation (i.e. strong consistency) then a Relational database (e.g. MySQL) is usually more suited than a Non-Relational Database (e.g.MongoDB).

Performance & Scaling

All databases' performance degrades as the amount of read/write traffic increases. This is the time when optimizations such as indexing your data and scaling your DB come into the picture.

Performance depends on various factors but the overall performance depends to a very large degree on choosing the right implementation for your use case.

SQL and NoSQL databases scale differently, so you’ll have to think about how your data set will grow in the future.

SQL databases scale vertically, meaning you’ll need to increase the capacity of a single server (increasing CPU, RAM, or SSD) to scale your database. SQL databases were designed to maintain the integrity of the data so they are best to run on a single server, hence they’re not easy to scale.

NoSQL databases are easy to scale horizontally, which means you can add more servers as your data grows. This is an advantage that NoSQL has over SQL.

This uncomplicated horizontal scaling nature of non-relational databases makes them superior to relational databases as far as availability is concerned.

The ability of NoSQL databases to scale horizontally has to do with the lack of structure of the data. Because NoSQL requires much less structure than SQL, each stored object is pretty much self-contained. Thus objects can be easily stored on multiple servers without having to be linked. This is not the case for SQL, where each table row and column needs to be related.

This is also the issue regarding performance and where SQL falls short is scaling. As a database grows in size and numbers, RDBMS as we have seen requires solutions in vertical scaling. This however comes at a significant cost. Although many commercial RDBMS products offer horizontal scaling, these can also be very expensive and even complex to implement.

If you predict you will face such an issue, then NoSQL is to be considered, as many of them were designed specifically to tackle these scale and performance issues.

Conclusion

In this article, we went through the multiple factors you should consider while selecting the type of database. In short, SQL databases provide great benefits for transactional data whose structure doesn’t change frequently (or at all) and where data integrity is paramount. It’s also best for fast analytical queries. NoSQL databases provide more flexibility and scalability, which makes them useful for rapid or iteration development.

Here's a small summary of what all we have discussed above:

Reasons to use an SQL database

• When you need ACID support
• Your application requires high transactions
• Data integrity is essential
• You don’t anticipate a lot of changes in the schema

Reasons to use a NoSQL database

• You want to store large amounts of data with no structure
• You keep getting Unrelated, indeterminate, or evolving data requirements
• Speed and scalability are critical
• When data integrity is not your top goal and you are concerned about the availability (& eventual consistency will be good enough for your use case)

I hope this helps you understand what you need to think about when selecting your database. What additional questions do you think about while selecting a database? Let me know your thoughts in the comments.

The 5 SOLID principles are:

• S - The single-responsibility principle
• O - The open-closed principle
• L - The Liskov substitution principle
• I - The interface segregation principle
• D - The dependency inversion principle

Although the SOLID principles apply to any programming language, in further section I will be explaining each of them with examples written specifically in JAVA.

Single Responsibility Principle

This principle states that “a class should have only one reason to change” which means every class should have a single responsibility.

public class Vehicle {
    public void details() {}
    public double price() {}
    public void addNewVehicle() {}
}

Here the class has multiple reasons to change because the Vehicle class has three separate responsibilities: printing details, printing price, and adding a new vehicle to Database.

To achieve the goal of the single responsibility principle, we should implement a separate class that performs a single functionality only.

public class VehicleDetails {
    public void details() {}
}

public class CalculateVehiclePrice {
    public double price() {}
}

public class AddVehicle {
    public void addNewVehicle() {}
}

Open-Closed Principle

This principle states that “software entities (classes etc.) should be open for extension, but closed for modification”. This means without modifying anything in a class, it should be extendable.

Let's understand this principle with an example of a notification service

public class NotificationService{
    public void sendNotification(String medium) {
         if (medium.equals("email")) {}
    }
}

Here, if you want to introduce a new medium other than email, let's say send a notification to a mobile number then you need to modify the source code in NotificationService class.

So to overcome this you need to design your code in such a way that everyone can reuse your feature by extending it and if they need any customization they can extend the class and add their feature on top of it.

You can create a new interface like:

public interface NotificationService {
    public void sendNotification(String medium);
}

Email Notification:

public class EmailNotification implements NotificationService {
    public void sendNotification(String medium){
        // write Logic using for sending email
    }
}

Mobile Notification:

public class MobileNotification implements NotificationService {
    public void sendNotification(String medium){
        // write Logic using for sending notification via mobile
    }
}

Liskov Substitution Principle

This principle states that “derived classes must be able to substitute for their base classes”. In other words, if class A is a child of class B, then we should be able to replace B with A without interrupting the current behavior of the program.

Consider an example of a square class derived from Rectangle base class:

public class Rectangle {
    private double height;
    private double width;
    public void setHeight(double h) { height = h; }
    public void setWidht(double w) { width = w; }
}
public class Square extends Rectangle {
    public void setHeight(double h) {
        super.setHeight(h);
        super.setWidth(h);
    }
    public void setWidth(double w) {
        super.setHeight(w);
        super.setWidth(w);
    }
}

In the Rectangle class, setting width and height seems perfectly logical. However, in the square class, the SetWidth() and SetHeight() don't make sense because setting one would change the other to match it.

In this case, Square fails the Liskov substitution test because you cannot replace the Rectangle base class with its derived class Square. The Square class has extra constraints, i.e., the height and width must be the same. Therefore, substituting Rectangle with Square class may result in unexpected behavior.

Interface Segregation Principle

This principle applies to Interfaces and it is similar to the single responsibility principle. It states that “ a client should never be forced to implement an interface that it doesn’t use, or clients shouldn’t be forced to depend on methods they do not use.“.

Let's understand this by the example of a vehicle interface:

public interface Vehicle {
    public void drive();
    public void stop();
    public void refuel();
    public void openDoors();
}

Let's say we now create a Bike class using this Vehicle interface

public class Bike implements Vehicle {
    public void drive() {}
    public void stop() {}
    public void refuel() {}

    // Can not be implemented
    public void openDoors() {}
}

Since Bike doesn't have doors we can't implement the last function.

To fix this, it is recommended to break down the interfaces into small multiple, interfaces so that no class is forced to implement any interface or methods, that it does not need.

public interface Vehicle {
    public void drive();
    public void stop();
    public void refuel();
}

public interface Doors{
    public void openDoors();
}

Creating two classes - Car and Bike

public class Bike implements Vehicle {
    public void drive() {}
    public void stop() {}
    public void refuel() {}
}

public class Car implements Vehicle, Door {
    public void drive() {}
    public void stop() {}
    public void refuel() {}
    public void openDoors() {}
}

Dependency Inversion Principle

The Dependency Inversion Principle (DIP) states that "entities must depend on abstractions (abstract classes and interfaces), and not on concrete implementations (classes). Also, the high-level module must not depend on the low-level module, but both should depend on abstractions".

Suppose there is a book store that enables customers to put their favorite books on a particular shelf.

In order to implement this functionality, we create a Book class and a Shelf class. The Book class will allow users to see reviews of each book they store on the shelves. The Shelf class will let them add a book to their shelf. For example,

public class Book {
    void seeReviews() {}
}


public class Shelf {
     Book book;
     void addBook(Book book) {}
}

Everything looks fine, but as the high-level Shelf class depends on the low-level Book, the above code violates the Dependency Inversion Principle. This becomes clear when the store asks us to enable customers to add their own reviews to the shelves, too. In order to fulfill the demand, we create a new UserReview class:

public class UserReview{
     void seeReviews() {}
}

Now, we should modify the Shelf class so that it can accept User Reviews, too. However, this would clearly break the Open/Closed Principle too.

The solution is to create an abstraction layer for the lower-level classes (Book and UserReview). We’ll do so by introducing the Product interface, both classes will implement it. For example, the below code demonstrates the concept.

public interface Product {
    void seeReviews();
}

public class Book implements Product {
    public void seeReviews() {}
}

public class UserReview implements Product {
    public void seeReviews() {}
}

Now, the Shelf can reference the Product interface instead of its implementations (Book and UserReview). The refactored code also allows us to later introduce new product types (for instance, Magazine) that customers can put on their shelves, too.

public class Shelf {
    Product product;
    void addProduct(Product product) {}
    void customizeShelf() {}
}

Conclusion

In this article, you were presented with the five principles of the SOLID Code. Adhering to SOLID principles can make your project, extendable, easily modifiable, well tested, with fewer complications.

Other Articles You Might Like

• Introduction to Asynchronous Processing and Message Queues - How to handle complex communication between a client and a server using message queues?

• Generating unique IDs in a Large scale Distributed environment - A decent way to generate unique IDs across a distributed system that could also be used as the primary keys in the MySQL tables.

• Javascript Clean Code Tips & Good Practices -15 tips I follow for writing better Javascript code.

• Five++ cool Python snippets that will blow your mind🤯 - Few clever yet useful tricks and tips that will surely make you think hard.

• Welcome to the world of "NFTs" - Learn about what are NFTs and Why are they suddenly becoming the next big thing.

In a client-server architecture, the client can request a job to be done from the server by sending messages to the server.

Handling this communication can increase in complexity when you begin to manage the rate at which messages are sent, the number of requests a server can handle, or the response time that the client demands.

In this blog, we'll see one way to handle this intricacy.

What is Synchronous Processing?

In synchronous processing, a client sends a request to the server and waits for the server to complete its job and send back the response before the client can resume doing any other work.

This process is often referred to as a blocking request, as the client gets blocked from doing any other work until a response is received.

What is Asynchronous Processing?

Asynchronous processing is the exact opposite of synchronous processing. Here, the client doesn't wait for a response after sending a request to the server and continues doing any other work.

This process is referred to as a non-blocking request, as the execution thread of the client is not blocked. This allows systems to scale as more work can be done in a given amount of time.

Synchronous vs Asynchronous Processing

• Synchronous requests block the execution thread of the client, forcing them to wait for the response to come before they can perform another action. On the other hand, asynchronous requests do not block and allow for more work to be done in a given time.

• Since we do not know how much time the request will take, it is difficult to build responsive applications with synchronous processing. The more blocking operations, the slower the system becomes. With asynchronous processing, response time is quicker as the client does not have to wait for the request.

• Fault tolerance of asynchronous processing is higher than that of synchronous processing, as it is easy to build a retry mechanism when a request fails.

What are Message Queues?

A message queue is a component that buffers requests from one service and broadcasts them asynchronously to another service.

Here, clients are the message producers, who send request messages to the queue instead of any server. They get an acknowledgment when the message is added to the queue, which enables them to continue with their other jobs without having to wait for the server.

Servers are known as message consumers and are served these messages from the queue based on the number of requests they can serve at any given time. This continues until all the messages in the queue have been served.

The two most common messaging queues are — RabbitMQ and Kafka.

Structure of Message Queues

A message queue is primarily a broker of messages between message producers and message consumers.

Each distinct entity in the messaging queue setup (producers, consumers, and queue) has a responsibility and they're decoupled from each other as much as possible.

The only contract between all entities is the messages for which the message queue facilitates the movement from producers to consumers.

In the following sections, we will discuss the responsibilities of each component and look at the various modes with which the message queue delivers a message to consumers.

Message Producers

Message producers initiate the asynchronous processing request. Producers have a responsibility to generate a valid message and publish it to the message queue. Messages submitted to the queue are then queued up and delivered to consumers to be processed asynchronously.

Producers communicate with message queues using the Advanced Message Queuing Protocol (AMQP).

Message Brokers

A message broker is simply just a queue. You can even implement a simple broker programmatically that buffers messages and sends them to consumers as and when needed.

Message brokers are the actual decoupling elements in the setup, sitting between and managing the process of communication between producers and consumers.

Because of their simplicity, brokers are optimized for high concurrency and throughput.

It is important to note that adding message brokers introduces an extra layer of complexity into your infrastructure and requires you to scale them as well. Brokers also have specific requirements and limitations when it comes to scalability.

Message Consumers

The main responsibility of consumers is to receive and process messages from the queue.

Most consumers are API services that perform that actual asynchronous processing.

Consumers can be implemented in different application languages or technologies and maintained independently from other components.

To achieve decoupling, consumers should know nothing about the producers. The only contract that should exist between the two is valid messages from the queue.

When properly decoupled, consumers can serve as independent service layers that can be used by both the message queue setup and other components in your infrastructure.

Consumer Communication Strategies

Message queues need to transmit messages down to consumers, depending on how application developers implement consumers, message queues have three distinct ways of delivering messages to the consumers:

• Pull Model

In this model, the consumer periodically checks the status of the queue. This is done at a scheduled interval programmed on the side of the consumer.

If there are messages found in the queue, the consumer picks them up until there are no more messages left to process, or when the 'N' number of messages has been consumed. This 'N' can be configured on the message broker.

• Push Model

Once a message is added, the consumer is notified and the message is then pushed down to it. Messages are pushed down to consumers at a rate at which the consumer can easily regulate.

• Subscription Model

In this model, consumers can subscribe to a topic. This publisher publishes a message to a topic rather than a queue. Each consumer connected to the broker maintains its private queue to receive messages from topics.

After the consumers subscribe to the topics and when a message is published to that topic, the message is cloned for each subscriber and added to the consumer’s private queue.

Comparing Different Message Brokers

As we have seen above, for asynchronous communication we usually need a message broker.

Below are the few considerations you have to look at when choosing a broker for managing your asynchronous operations:

• Scale: The number of messages sent per second in the system

• Data Persistency: The capability to recover messages

• Consumer Capability: The capability to manage one-to-one / one-to-many consumers

RabbitMQ

• Scale: Based on configuration and resources.

• Persistency: Both persistent and transient messages are supported.

• One-to-one vs One-to-many consumers: Both.

RabbitMQ supports all major languages, including Python, Java, .NET, PHP, Ruby, JavaScript, Go, Swift, and more.

Kafka

• Scale: Can send up to a million messages per second.

• Persistency: Yes.

• One-to-one vs One-to-many consumers: Only one-to-many

Kafka has managed SaaS on both Azure and AWS. Kafka also supports all major languages, including Python, Java, C/C++, Clojure, .NET, PHP, Ruby, JavaScript, Go, Swift, and more.

Redis

• Scale: Can send up to a million messages per second.

• Persistency: Not supported (it’s an in-memory database).

• One-to-one vs One-to-many consumers: Both.

Redis is a bit different from the other message brokers. Redis is an in-memory data store. Originally, Redis only supported one-to-one communication with consumers. However, since Redis 5.0 introduced the pub-sub, you can have one-to-many as another option.

Conclusion

In this blog, we covered how asynchronous processing is advantageous over its counterparts and how a message queue helps us achieve asynchronous processing along with keeping the different entities in its setup decoupled from each other.

We also covered some basic characteristics of the most commonly used message brokers: Redis, Kafka, and RabbitMQ.

Here's a bit more instruction for selecting the right message broker to use according to different use cases:

Short-lived Messages: Redis

• Redis is good for short-lived messages where persistence isn’t required.

Large Amounts of Data: Kafka

• Kafka is good for storing a large amount of data for long periods. Kafka is also ideal for one-to-many use cases where persistency is required.

Complex Routing: RabbitMQ

• RabbitMQ is good for complex routing communication.

Happy coding! 💻

(If you find any doubts, updates, or corrections to improve this article, Feel free to share them in the comments) 😊

Other Articles You Might Like

• Generating unique IDs in a Large scale Distributed environment - A decent way to generate unique IDs across a distributed system that could also be used as the primary keys in the MySQL tables.

• Javascript Clean Code Tips & Good Practices -15 tips I follow for writing better Javascript code.

• How we solved cache invalidation in Kubernetes with a headless service - A clever way we engineered to solve one of the finest computer science problems.

• Welcome to the world of "NFTs" - Learn about what are NFTs and Why are they suddenly becoming the next big thing.

• What is Blockchain Technology? Learn by creating one - Want to know how blockchain works?

In this article, you will learn how to use Spring Boot and HarperDB to create a microservice. Later on, you will also look at how to deploy the complete application on AWS Elastic Beanstalk.

You will be building an Employee Leave Management System. This application will be responsible for tracking the detailed record of employees' leaves. You will also be implementing the functionality to add, edit, and cancel leaves.

But first, let's get a basic understanding of microservices.

What are Microservices?

Microservices are another design architecture for developing software. In this, the software is composed of small independent services that communicate over REST APIs.

Microservices architectures make applications easier to develop and scale. It also enables an organization to evolve its technology stack easily later on if required.

Monolithic vs. Microservices

In monolithic architectures, all processes run as a single service, therefore, they are tightly coupled. This means that if one process goes down, the entire application will get affected leading to a single point of failure.

Also, enhancing or adding new features becomes more complicated as the code base grows. This complexity makes it difficult to experiment and implement new ideas.

In microservices architectures, an application is built by combining different components that run independently as a service. These services communicate over a well-defined interface of lightweight REST APIs.

Each service is responsible for performing a single task and because they run independently, each service can be updated, deployed, and scaled to meet the demand for specific functions of an application.

Introducing HarperDB

You are going to use HarperDB as your Database. HarperDB is a complete data management solution and distributed database with native SQL operations like join, order by, etc., and NoSQL schema-less or even API-based execution.

The most notable features of HarperDB are:

• It has one endpoint for all CRUD operations
• Execute SQL queries on JSON data
• Supports multiple plugins like ODBC, JDBC, Node-RED, etc.
• It has both SQL and NoSQL support
• Eliminates the need for an ORM(Object Relational Mapping) by returning the results as arrays of JSON
• Executes complex and ACID-compliant SQL queries on JSON without any data duplication

Looks interesting, doesn't it?

Configure HarperDB Database Instance

Let's now start with configuring the HarperDB instance:

• Go to https://studio.harperdb.io/sign-up

• Fill out the form with your details

• Click Sign Up For Free

• You will then receive an email that looks like this:

• Click on Log into HarperDB studio

Creating an Instance

• Click on Create HarperDB Cloud Instance

• Enter Your Credentials

Note: These credentials should NOT be confused with your Studio account. This is for the superuser you are creating on this database instance. The username should NOT be in the form of an email.

• Pick your instance specs. You can start with a free version and upgrade later as per your need.

• Next, review your details and click Confirm Instance Details

• Finally, Click on Add Instance

In another 5 to 10 minutes your instance will be ready. You'll also receive an email as soon as it's done.

The instance URL that you receive is how you can access HarperDB with REST calls.

Initially, your instance does not have any schemas or tables. So you need to create them first.

I chose Employee_Leaves as the name for my schema. However, you can use any other name for your schema and then click on the green checkmark to save it.

Next, you need to create tables. For the moment, you'll set up two tables Employee and Leaves, the first one is for storing all the employee details like names and employee Ids and the second one is to store the leaves of employees in the company.

Note that you do not need to add all the columns just yet, they will be added automatically when needed. For now, while creating a table you just have to provide a hash_attribute name. The hash attribute is used to uniquely identify each record. For both these tables, you can use ID as hash_attribute.

Creating a Spring Boot App

To start with spring boot, you have to create a basic application from Spring.io.

Select Maven project and Java language. For the Spring Boot version, select 2.5.3. You will also have to add the Spring-boot-starter-web and Lombok dependency.

Optionally, fill in the project metadata. For instance, you can set group as com.employee, artifact & name as attendance, package as com.employee.attendance, and finally enter a short description & click on Generate.

Extract the downloaded project and open it in your favorite IDE.

Next, for accessing the HarperDB instance from your Spring Boot app you first have to download the CData JDBC Driver for HarperDB.

The CData JDBC driver allows the most straightforward way to connect to HarperDB from any Java-based application. It wraps and hides the complexity of accessing data and provides additional powerful security features, smart caching, batching, socket management, and more.

Let's see the steps for integrate this driver inside your application:

• Navigate to - https://studio.harperdb.io/resources/drivers and then click on download.

• Go back to your project root directory and create a new folder called lib & extract the zip file contents inside this folder.

• Next, to import the external jar into your spring boot app, open the pom.xml file and add the following dependency in it:

<dependency>
    <groupId>cdata.jdbc.harperdb</groupId>
    <artifactId>cdata.jdbc.harperdb</artifactId>
    <scope>system</scope>
    <version>1.0</version>
    <systemPath>${project.basedir}/lib/cdata.jdbc.harperdb.jar</systemPath>
</dependency>

Here, the ${project.basedir} refers to the path where your pom.xml is saved i.e your root directory. If you have the driver installed at any other path, you have to add its absolute path inside the <systemPath> Tag.

An important thing to note here is that you have to set the <scope> as system and also, inside the plugins set the <includeSystemScope> as true.

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <configuration>
    <includeSystemScope>true</includeSystemScope>
  </configuration>
</plugin>

This has to be done since you are ultimately going to deploy this application on a server(AWS) and for a server to know about any external JAR files, these configurations are necessary.

Now, to establish a connection between your app and the HarperDB instance, create a new package and name it service. Inside this, make a new Java file called ConnectionService.java and add the following content to it:

package com.employee.attendance.service;

import org.springframework.stereotype.Service;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

@Service
public class ConnectionService {
    public Connection createConnection() throws SQLException {
        return DriverManager.getConnection("jdbc:harperdb:Server=https:xyz.harperdbcloud.com;User=admin;Password=1234;UseSSL=true;");
    }
}

@Service annotation is used with classes that provide some business functionalities

Here, you are making the connection with the Database using the "Connection String" inside the getConnection() method available under the DriverManager class.

Alternatively, you can also prepare the connection options using a Properties object. Pass the Properties object to the DriverManager:

Properties prop = new Properties();
prop.setProperty("Server","https:xyz.harperdbcloud.com");
prop.setProperty("User","admin");
prop.setProperty("Password","1234");

Connection conn = DriverManager.getConnection("jdbc:harperdb:",prop);

Replace the server, user, and password with your own credentials and you are good to go.

Designing REST APIs

Now that a connection has been successfully established between your app and HarperDB, let's begin with creating the Restful APIs.

First, create a new package called controller. Inside this, create a new class and name it AttendanceController.java.

A controller class is where you will expose all the endpoints of your microservices. These endpoints are what different microservices use to communicate with each other.

For this application, you will create 4 endpoints:

• GET /api/get/all/leaves/{employeeId} - To get all the leaves that a particular employee has applied so far.

• POST /api/add/leave - To add a new leave for an employee.

• PUT /api/edit/leave - To edit the date of an existing leave.

• DELETE /api/cancel/leave - To delete an existing leave.

Here's how the complete controller class will look after adding these 4 endpoints:

package com.employee.attendance.controller;

import com.employee.attendance.dto.EmployeeDataDTO;
import com.employee.attendance.dto.EmployeeEditDataDTO;
import com.employee.attendance.service.AttendanceService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.util.HashMap;
import java.util.List;

@RestController
public class AttendanceController {

    @Autowired
    private AttendanceService service;

    @GetMapping(value = "/api/get/all/leaves/{employeeId}")
    public List<HashMap<String, String>> getAllLeavesForEmployee(@PathVariable String employeeId) {
        return service.getAllLeavesForEmployee(employeeId);
    }

    @PostMapping(value = "/api/add/leave")
    public HashMap<String, String> addNewLeave(@RequestBody EmployeeDataDTO employeeData) {
        return service.addNewLeaveForEmployee(employeeData);
    }

    @PutMapping(value = "/api/edit/leave")
    public HashMap<String, String> editLeave(@RequestBody EmployeeEditDataDTO employeeEditData) {
        return service.editLeaveForEmployee(employeeEditData);
    }

    @DeleteMapping(value = "/api/cancel/leave")
    public HashMap<String, String> cancelLeave(@RequestBody EmployeeDataDTO employeeData) {
        return service.cancelLeaveForEmployee(employeeData);
    }
}

Let us now create another class called AttendanceService.java in the service package that you created earlier. Inside this class, you will write all your business logic. So, let's implement all 4 methods which are mentioned in your controller:

package com.employee.attendance.service;

import com.employee.attendance.dto.EmployeeDataDTO;
import com.employee.attendance.dto.EmployeeEditDataDTO;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.sql.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

@Service
@Slf4j
public class AttendanceService {

    @Autowired
    private ConnectionService connectionService;

    public List<HashMap<String, String>> getAllLeavesForEmployee(String empId) {
        log.info("Getting all leaves for employee - {}",empId);
        List<HashMap<String, String>> resultList = new ArrayList<>();
        try {
            Connection conn = connectionService.createConnection();
            PreparedStatement statement = conn.prepareStatement("Select * From Employee_Leaves.Leaves where empId = ?");
            statement.setString(1,empId);
            ResultSet resultSet = statement.executeQuery();
            while (resultSet.next()) {
                HashMap<String, String> result = new HashMap<>();
                result.put("date_of_apply",new Date(Long.parseLong(resultSet.getString("__createdtime__"))).toString());
                result.put("last_update_date",new Date(Long.parseLong(resultSet.getString("__updatedtime__"))).toString());
                result.put("leave_applied_for",resultSet.getString("date"));
                result.put("employee_id",resultSet.getString("empId"));
                resultList.add(result);
            }
            conn.close();
        } catch (Exception e){
            log.error("Error occurred", e);
            HashMap<String, String> result = new HashMap<>();
            result.put("Error",e.getMessage());
            resultList.add(result);
        }
        return resultList;
    }

    public HashMap<String, String> addNewLeaveForEmployee(EmployeeDataDTO employeeData) {
        log.info("Inserting new leave for employee - {}",employeeData.getEmployeeId());
        HashMap<String, String> result = new HashMap<>();
        try {
            Connection conn = connectionService.createConnection();
            PreparedStatement statement = conn.prepareStatement("INSERT INTO Employee_Leaves.Leaves (date, empId) VALUES (?,?)");
            statement.setString(1, employeeData.getDate());
            statement.setString(2, employeeData.getEmployeeId());
            int count = statement.executeUpdate();
            if(count>0) {
                result.put("Message", "Success");
                result.put("Affected rows", String.valueOf(count));
            }
            conn.close();
        } catch (Exception e){
            log.error("Error occurred", e);
            result.put("Error",e.getMessage());
        }
        return result;
    }

    public HashMap<String, String> editLeaveForEmployee(EmployeeEditDataDTO employeeEditData) {
        log.info("Updating leave for employee - {}",employeeEditData.getEmployeeId());
        HashMap<String, String> result = new HashMap<>();
        try {
            Connection conn = connectionService.createConnection();
            PreparedStatement statement = conn.prepareStatement("UPDATE Employee_Leaves.Leaves SET date = ? WHERE empId=? and date = ?");
            statement.setString(1, employeeEditData.getNewDate());
            statement.setString(2, employeeEditData.getEmployeeId());
            statement.setString(3, employeeEditData.getPreviousDate());
            int count = statement.executeUpdate();
            if(count>0) {
                result.put("Message", "Success");
                result.put("Affected rows", String.valueOf(count));
            }
            conn.close();
        } catch (Exception e){
            log.error("Error occurred", e);
            result.put("Error",e.getMessage());
        }
        return result;
    }

    public HashMap<String, String> cancelLeaveForEmployee(EmployeeDataDTO employeeData) {
        log.info("Cancelling leave for employee - {}",employeeData.getEmployeeId());
        HashMap<String, String> result = new HashMap<>();
        try {
            Connection conn = connectionService.createConnection();
            PreparedStatement statement = conn.prepareStatement("DELETE FROM Employee_Leaves.Leaves WHERE empId = ? and date = ?");
            statement.setString(1, employeeData.getEmployeeId());
            statement.setString(2, employeeData.getDate());
            int count = statement.executeUpdate();
            if(count>0) {
                result.put("Message", "Success");
                result.put("Affected rows", String.valueOf(count));
            }
            conn.close();
        } catch (Exception e){
            log.error("Error occurred", e);
            result.put("Error",e.getMessage());
        }
        return result;
    }
}

With that done, let's understand what each of these four functions is used for:

• getAllLeavesForEmployee().

It will be responsible for fetching all the leaves for a particular employee from the Leaves table. You are simply establishing a connection and then querying the Leaves table based on employee Id.

• addNewLeaveForEmployee()

Whenever an employee needs to apply for a leave, this function will be invoked which will take an Object of EmployeeDataDTO class and will be adding a new leave inside the Leave table of your database.

• editLeaveForEmployee()

If in case an employee wants to change the date for which leave has been applied before, the editLeaveForEmployee() function will be responsible for modifying that in your Database.

• cancelLeaveForEmployee()

It will simply cancel a particular leave for an employee i.e. Delete the row from the table. This function also takes the object of the EmployeeDataDTO class as its parameters.

Next thing is to understand the use of EmployeeDataDTO and EmployeeEditDataDTO classes.

These two classes will act as your Data Transfer Objects or simply DTOs. They will store the data which you will pass in the API request body & carry that to the service for processing via the controller class.

To generate these Java files, create a new package called dto and add two files in it, one will be EmployeeDataDTO.java and another one will be EmployeeEditDataDTO.java.

Here's the content EmployeeDataDTO.java will hold:

• Employee Id
• Date for which leave is being applied

package com.employee.attendance.dto;

import lombok.Data;

@Data
public class EmployeeDataDTO {
    private String employeeId;
    private String date;
}

And EmployeeEditDataDTO.java will have the following properties:

• Employee Id
• Previous Date which user wants to edit
• New Date for Leave

package com.employee.attendance.dto;

import lombok.Data;

@Data
public class EmployeeEditDataDTO {
    private String employeeId;
    private String previousDate;
    private String newDate;
}

@Data is a useful annotation that bundles the features of @ToString, @EqualsAndHashCode, @Getter / @Setter. It is available under Lombok dependency.

Testing the Endpoints

I have manually inserted some initial data into the database tables (by clicking the '+' sign on the top right of the table) for testing purposes:

• Table Employee

• Table Leave

Now you have to hit those 4 API endpoints to check if everything is working as expected.

For this, you must download the Postman app in your local development environment from their official downloads.

After downloading and successfully installing it, open the app and run your Spring Boot application.

In case you are using an IDE like IntelliJ IDEA, you can run the Spring Boot application by following these instructions:

• Click on Add Configuration from the top menu bar.

• A new dialog box will open, click on Add new run configurations and select Maven from the dropdown.

• Give your run configuration a name. The working directory will automatically get picked. You just have to put the command spring-boot:run inside the command line and then click Apply and OK and finally run the app.

The next thing you have to do is to hit the endpoints via the Postman app:

• /api/get/all/leaves/{employeeId}

• /api/add/leave

To check if the leave got added successfully or not, let's call the api/get/all/leaves API again which you used in the first step:

As you can see in the response, the employee whose ID is 1 now has two leaves. This same has been reflected in the Database:

• /api/edit/leave

Again, let's check if the previous date has been modified or not, you will get:

This means that the edit functionality is also working fine.

• /api/cancel/leave

And now hitting the GET leaves API, you should only get one entry in response:

So, all your endpoints are working as expected in the local development environment. Let's move forward to see how you can deploy these APIs to the server.

Creating a JAR file

A JAR(Java Archive) is a package file format used to cluster together all the Java class files and associated metadata and resources (text, images, etc.) into one file to distribute application software or libraries on the Java platform.

In simple terms, a JAR file contains a compressed version of .class files, audio files, image files, or other directories.

To create a JAR file, make sure Maven is installed in your local development environment. If not, follow the below steps to configure maven (In Windows OS):

• Navigate to the official Maven website and download the Maven zip file. For example: apache-maven-3.8.2-src.zip.

• Unzip the contents and copy the path of the bin folder inside.

• Open your System Environment Variables from the control panel, find the PATH variable then click on the Edit button.

• In the "Edit environment variable" dialog, click on New and add the complete path of the bin folder that you just copied.

• Finally, click on OK. To test the configuration, open a new command prompt and type mvn –version. If a version is visible to you that means the configurations are correct.

If you're using any other OS, you can find the installation steps for that on the official Maven website. For more detailed steps, you can check out this article as well.

Next, to create a JAR file for your application run the command mvn clean install in your project's root directory.

A new folder with the name Target will be created. Inside this folder, you'll find your newly created JAR file.

Another way to create a JAR file is from your IDE. For instance, in Intellij you can navigate to the maven tab from the sidebar menu and double click Install.

And here's how your complete project structure should look like by now:

├── lib
├── src
├── target
└── pom.xml

Inside src>main>java>com>employee>attendance you should have the following files and structure:

├── controller
│   └── AttendanceController.java
├── dto
│   └── EmployeeDataDTO.java
│   └── EmployeeEditDataDTO.java
├── service
│   └── AttendanceService.java
│   └── ConnectionService.java
└── AttendanceApplication.java

Deploying the application to AWS

The final step would be to deploy your code to a server. In this section, you'll learn how you can deploy the application to AWS Elastic Beanstalk.

Using Elastic Beanstalk, you can easily deploy and manage applications in the AWS Cloud without worrying about the infrastructure that runs those applications. It reduces management complexity without restricting choice or control. You simply upload your application, and Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring.

Creating Elastic Beanstalk environment

Once you are signed in to your AWS account, go to the search panel at the top and type "Elastic Beanstalk," and click create a new application on the top right.

It will ask for your:

• Application name
• Application tags(Not mandatory)
• Platform
• Application code

Enter your application name and optionally you can add up to 50 tags for the resources of your Elastic Beanstalk applications.

For the platform, select "Java" from the dropdown and it will automatically fill the "platform branch" and "version" on its own.

For the application code, select Upload your code and then choose the JAR file built in a previous step.

Review the configuration and launch the environment. As the application is being launched, you’ll see something similar to this on the environment dashboard:

Once the application resources get deployed and the environment has been created, you’ll notice the health of the application remains Severe. This is because the Spring Boot application still needs some configuration:

Let's look at how to solve it in the next section.

Configuring Spring Boot Through Environment Variables

By default, Spring Boot applications listen on port 8080 whereas Elastic Beanstalk expects the application to listen on port 5000.

There are two ways to fix this error, either you have to change the port Elastic Beanstalk is configured to use, or change the port the Spring Boot application listens on. For this tutorial, you will change the port the Spring Boot application listens on as it is easier.

For this, you have to define the SERVER_PORT environment variable in the Elastic Beanstalk's environment and set its value to 5000.

• Click on the Configuration from the sidebar menu on your environment page.

• On the Configuration page, you'll have software configuration, click on Edit:

• Next, you’ll see that there are already some environment variables set. They are set automatically by Elastic Beanstalk when it is configured to use the Java platform.

• To modify the port that Spring Boot listens on, as mentioned above you need to add a new environment variable by the name SERVER_PORT with its value 5000.

Click on Apply and the configuration change will make the application restart.

After it restarts, it will pick up the new configuration through the environment variables. In about a minute, you’ll get a healthy application instance up and running.

Testing the Application in the Cloud

Once deployed successfully, you will get your application base URL. You can also access the same from the sidebar:

Now replace the localhost:8080 with this new URL and try to access one of your API endpoints:

And you can get the correct response. This means that the application has been deployed successfully.

Conclusion

With this, you come to the end of this tutorial. The purpose of this tutorial was to use Spring Boot and HarperDB to create microservices. You also looked at the step-by-step process of how you can deploy the same application to AWS.

Both Spring Boot and AWS are used extensively in the industry but HarperDB might be new for you. The reason for choosing HarperDB is because of its easy integration with the Spring Boot app along with the notable features that I have mentioned at the beginning of this article.

I hope you learned something new. In case you get stuck somewhere, please feel free to shoot your doubts in the comments.

Also, the complete source code of the application used in this tutorial can be found in this GitHub Repository.

The more readable our source code is:

• The easier it is to maintain
• The less time required to understand an implementation for a new developer
• The easier it is to discover what code can be reused

In this blog post, I will share some general clean coding principles that I've adopted over time as well as some JavaScript-specific clean code practices.

0. Naming

Don't turn naming into a riddle game. Name your variables and functions in a way that they reveal the intention behind why they were created in the first place.

This way they become searchable and easier to understand if let's say a new developer joins the team.

Only go for Shortening and abbreviating names when you want the next developer working on your code to guess what you were thinking about 😉

Bad 👎

let x = 10;

let y = new Date().getFullYear();

if (x > 30) {
    //...
}

if (y - x >1990) {
    //...
}

Good 👍

let userAge = 30;

let currentYear = new Date().getFullYear();

if (userAge > 30) {
    //...
}

if (currentYear - userAge >1990) {
    //...
}

Also, don’t add extra unnecessary letters to the variable or functions names.

Bad 👎

let nameValue;
function theProduct();

Good 👍

let name;
function product();

1. Conditionals

Avoid negative conditionals. Negatives are just a bit harder to understand than positives.

Bad 👎

if (!userExist(user)) {
  //...
}

Good 👍

if (userExist(user)) {
  //...
}

2. Functions should do one thing

The function should not have more than an average of 30 lines (excluding spaces and comments). The smaller the function the better it is to understand and refactor. Try making sure your function is either modifying or querying something but not both.

3. Use default arguments

Use default arguments instead of short-circuiting or conditionals.

Default arguments are often cleaner than short-circuiting. Remember that if you use them, your function will only provide default values for undefined arguments. Other falsy values such as '', "", false, null, 0, and NaN, will not be replaced by a default value.

Bad 👎

function getUserData(name) {
  const userName = userName || "Patrick Collision";
  // ...
}

Good 👍

function getUserData(name = "Patrick Collision") {
  // ...
}

4. Single Level of Abstraction(SLA)

While writing any function, if you have more than one level of abstraction, your function is usually doing more than one thing. Dividing a bigger function into multiple functions leads to reusability and easier testing.

Functions should do one thing. They should do it well. They should do it only. — Robert C. Martin

Bad 👎

function checkSomething(statement) {
  const REGEXES = [
    // ...
  ];

  const statements = statement.split(" ");
  const tokens = [];
  REGEXES.forEach(REGEX => {
    statements.forEach(statement => {
      // ...
    });
  });

  const names= [];
  tokens.forEach(token => {
    // lex...
  });

  names.forEach(node => {
    // parse...
  });
}

Good 👍

function checkSomething(statement) {
  const tokens = tokenize(statement);
  const syntaxTree = parse(tokens);
  syntaxTree.forEach(node => {
    // parse...
  });
}

function tokenize(code) {
  const REGEXES = [
    // ...
  ];

  const statements = code.split(" ");
  const tokens = [];
  REGEXES.forEach(REGEX => {
    statements.forEach(statement => {
      tokens.push(/* ... */);
    });
  });

  return tokens;
}

function parse(tokens) {
  const syntaxTree = [];
  tokens.forEach(token => {
    syntaxTree.push(/* ... */);
  });

  return syntaxTree;
}

5. Don't ignore caught errors

Doing nothing with a caught error doesn't give you the ability to fix or react to that particular error.

Logging the error to the console (console.log) isn't much better as oftentimes it can get lost among other things printed to the console.

If you wrap any bit of code in a try/catch it means you think an error may occur there and therefore you should have a plan for when it occurs.

Bad 👎

try {
  functionThatMightThrow();
} catch (error) {
  console.log(error);
}

Good 👍

try {
  functionThatMightThrow();
} catch (error) {
  notifyUserOfError(error);   
  reportErrorToService(error);   
}

6. Minimize Comments

Only comment the part of the code that has business logic complexity. Comments are not a requirement. Good code mostly documents itself.

Bad 👎

function hashing(data) {
  // The hash
  let hash = 0;

  // Length of string
  const length = data.length;

  // Loop through every character in data
  for (let i = 0; i < length; i++) {
    // Get character code.
    const char = data.charCodeAt(i);
    // Make the hash
    hash = (hash << 5) - hash + char;
    // Convert to 32-bit integer
    hash &= hash;
  }
}

Good 👍

function hashing(data) {
  let hash = 0;
  const length = data.length;

  for (let i = 0; i < length; i++) {
    const char = data.charCodeAt(i);
    hash = (hash << 5) - hash + char;

    // Convert to 32-bit integer
    hash &= hash;
  }
}

“Redundant comments are just places to collect lies and misinformation.” ― Robert C. Martin

7. Remove commented code

Don't leave commented out code in your codebase, Version control exists for a reason. Leave old code in your history. If you ever need them back, pick them up from your git history.

Bad 👎

doSomething();
// doOtherStuff();
// doSomeMoreStuff();
// doSoMuchStuff();

Good 👍

doSomething();

8. Import only what you need

Destructuring was introduced with ES6. It makes it possible to unpack values from arrays, or properties from objects, into distinct variables. You can use this for any kind of object or module.

For instance, if you only require to add() and subtract() function from another module:

Bad 👎

const calculate = require('./calculations')

calculate.add(4,2);
calculate.subtract(4,2);

Good 👍

const { add, subtract } = require('./calculations')

add(4,2);
subtract(4,2);

It makes sense to only import the functions you need to use in your file instead of the whole module, and then access the specific functions from it.

9. Keep Function arguments 3 or less (ideally)

Limiting the number of function parameters is really important because it makes testing your function easier. Having more than three parameters leads you to test tons of different cases with each separate argument.

1-3 arguments are the ideal case, anything above that should be avoided if possible.

Usually, if you have more than three arguments then your function is trying to do too much. Which ultimately leads to the violation of the SRP(Single Responsibility Principle).

10. Use array spreads to copy arrays.

Bad 👎

const len = items.length;
const itemsCopy = [];
let i;

for (i = 0; i < len; i += 1) {
  itemsCopy[i] = items[i];
}

Good 👍

const itemsCopy = [...items];

11. Write linear code

Nested code is hard to understand. Always write the linear code as much as possible. It makes our code simple, clean, easy to read, and maintain, thus making developer life easier.

For Example, Using promises over callbacks can increase readability multiple times.

12. Use ESLint and Prettier

Always use ESLint and Prettier to enforce common coding styles across teams and developers.

Also try and use JavaScript's latest features to write code, like destructuring, spread operator, async-await, template literals, optional chaining, and more.

13. Use proper parentheses

When working with operators, enclose them in parentheses. The only exception is the standard arithmetic operators: +, -, and * since their precedence is broadly understood. It is highly recommended to enclose /, , and % in parentheses because their precedence can be ambiguous when they are used together.

This improves readability and clarifies the developer’s intention.

Bad 👎

const foo = a && b < 0 || c > 0 || d + 1 === 0;

if (a || b && c) {
  return d;
}

Good 👍

const foo = (a && b < 0) || c > 0 || (d + 1 === 0);

if (a || (b && c)) {
  return d;
}

Make sure your code doesn't lead to situations like this:

14. Return early from functions

To avoid deep nesting of if-statements, always return a function's value as early as possible.

Bad 👎

function isPercentage(val) {
  if (val >= 0) {
    if (val < 100) {
      return true;
    } else {
      return false;
    }
  } else {
    return false;
  }
}

Good 👍

function isPercentage(val) {
  if (val < 0) {
    return false;
  }

  if (val > 100) {
    return false;
  }

  return true;
}

This particular example can even improve further:

function isPercentage(val) {
  var isInRange = (val >= 0 && val <= 100);
  return isInRange;
}

Similarly, the same thing can be applied to Loops as well.

Looping over large cycles can surely consume a lot of time. That is why you should always try to break out of a loop as early as possible.

Conclusion

There’s a saying in the development community that you should always write your code like the next developer that comes after you is a serial killer.

Following this rule, I have shared 15 tips here that can (probably) save you from your fellow developers when they will look into your code.

If you find any updates or corrections to improve these 15 tips or want to add one of your own that you think can be helpful, please feel free to share them in the comments.

For further reading I would highly suggest you go through these 3 resources:

• Airbnb JS style guide
• Google JS style guide
• Javascript Clean Code

In every software system, we need unique Ids to distinguish between several objects from one another.

Recently, I wrote about the unique Id generation for a large scale distributed environment. In that article, we touched a little on UUIDs.

In this blog post, I will share an alternative to UUIDs that can help you fit the requirements that UUIDs fulfills and overcomes some of their shortcomings as well.

Introducing Nano ID

It is a tiny, secure, URL-friendly, unique string ID generator.

• Nano ID has smaller size as compared to UUID. This size reduction impacts a lot. Making use of NanoID is easier for transferring information and storage space. In a large-scale system, these numbers can make a lot of difference.

• NanoID uses a cryptographically strong API which is a lot safer compared to Math.Random() which are insecure. These API modules use unpredictable hardware generated random identifiers.

• NanoID utilizes its very own "uniform formula" throughout the application of the ID generator instead of making use of an arbitrary % alphabet which is a popular mistake to make when coding an ID generator (The spread will not be even in some cases).

• NanoID uses a larger alphabet resulting in short but unique identifiers.

• NanoID permits designers to utilize personalized alphabets. This is another additional function of Nano ID. You can alter the literals or the dimension of the id as shown below (Specifying personalized letter as '1234567890ABCDEF' & dimension of the Id as 10):

import { alphabet } from 'nanoid';
const nanoid = alphabet ('1234567890ABCDEF', 10);
model.id = nanoid();

• NanoID doesn’t much rely on any kind of third-party dependencies, which means, it ends up being a lot more steady which is helpful to maximize the package scope over time as well as make it much less vulnerable to the issues that come along with dependencies.

• NanoID is available in various programs languages, which include - C#, C++, Dart & Flutter, Go, Java, PHP, Python, Ruby, Rust, Swift, etc.

Benchmark

Example usages

Generating both NanoID or UUID is pretty straightforward. In JavaScript, you have NPM packages that will help you to generate them. You can get NanoId from here => https://github.com/ai/nanoid

• The main module uses URL-friendly symbols (A-Za-z0-9_-) and returns an ID with 21 characters:

import { nanoid } from "nanoid";
model.id = nanoid() // X2JaSYP7_Q2leGI9b-MyA

• You can also specify the number of characters you want:

nanoid(9); // "wMeKBp6th"

• You can also change the used alphabet for generating hashes to your own if you have specific requirements as seen above:

const alphabet = '0123456789ABCDEF';
generate(alphabet, 9); // F65BF3050

Risk of collision

Even though it can generate over 2.2 million unique IDs per second with its default alphabet, there is still a chance of generating the same multiple Ids.

But what are the odds of that happening?

You can calculate that based on the given parameters easily here and here.

You'll notice that this probability comes out to be extremely small.

Some disadvantages

• Being non-human readable can be one of the disadvantages in some cases

Imagine that a customer calls and is asked to provide the identifier, having to spell a complete NanoID is not a pleasant experience. When compared to UUID, NanoID is way shorter and readable but still cannot be used in such cases where the end customer needs to use it.

• It can't be used as primary key in any SQL Database tables

If you use NanoID as a table’s primary key, there will be problems if you use the same column as a clustered index. This is because NanoIDs are not sequential.

This is because the nature of NanoID(or even UUID) is that it's random and a clustered index physically orders the records by the key, so for every insert if there are indexes on the table, the database must make sure the new entry is also found via these indexes and to keep the index order and tree balance.

For this SQL has to reorder the records on disk and therefore remove clustering from this index. But when you have something sequential like time - clustering is almost free and easy to do after inserting a new record.

A few words of wisdom

Any approach in a Software World is always going to be subjective. It’s up to your requirements to weigh in the tradeoffs and choose the approach that works for you. No design is concrete enough to continue forever, so given the constraints, we have chosen a certain design, and depending on how it works for us we might evolve it further as well.

👋 Thanks for reading and Happy learning…

1. Use Numeric Separators

This is one of the most used operators when I have to deal with large numbers. When using a separator (with just an _) in number it looks better than an unseparated number.

For example:

let number = 98234567

to ⬇

let number = 98_234_567

And it works for any other numeric base as well:

const binary = 0b1000_0101;
const hex = 0x12_34_56_78;

Few caveats :

• More than one underscore in a row is not allowed let num= 100__00

• Can not be used after leading 0 let num= 0_1

• Not allowed at the end of numeric literals let num= 100_

2. Always Use Semicolons

The use of semi-colons for line termination is a good practice. You won’t be warned if you forget it, because in most cases it will be inserted by the JavaScript parser but relying on Automatic Semicolon Insertion(ASI) is not encouraged.

This is even included in Google’s, Airbnb’s, and jQuery’s Javascript style guides.

To know about what could happen if we rely too much on ASI, checkout the 4th issue of my newsletter I shared some months back. In the last section, I have explained it with an example.

3. Don’t forget "var"

When you assign a variable’s value for the first time, always make sure you're not doing it to an undeclared variable.

Assignment to an undeclared variable automatically results in a global variable being created. Avoid global variables ❌

Global variables are easily overwritten by other scripts. For example, if two separate parts of an application define global variables with the same name but with different purposes, it can result in unpredicted errors and it will be a horrible experience to debug such a problem.

Generally, you should try to scope your code so that you need as little as possible in the global scope. The more global variables you use in your script, the less is the chance that you can use it alongside another script.

Normally variables in a function should be local so that they go away when you exit the function.

"By reducing your global footprint to a single name, you significantly reduce the chance of bad interactions with other applications, widgets, or libraries." - Douglas Crockford

4. Delete vs Splice

Use splice instead of using delete to remove an item from an array. Using delete will remove the object property, but will not reindex the array or update its length. This makes it appears as if it is undefined.

Delete

> myArray = ['a', 'b', 'c', 'd']
  ["a", "b", "c", "d"]
> delete myArray[0]
  true
> myArray[0]
  undefined

Note that it is not in fact set to the value undefined, rather the property is removed from the array, making it appear undefined. The Chrome dev tools make this distinction clear by printing empty when logging the array.

Splice

Splice() actually removes the element, reindexes the array, and changes its length.

> myArray = ['a', 'b', 'c', 'd']
  ["a", "b", "c", "d"]
> myArray.splice(0, 2)
  ["a", "b"]
> myArray
  ["c", "d"]

The delete method should be used to delete an object property.

5. map vs for loop

Although there are various ways to loop through an array, use the map() function method whenever possible.

var squares = [1,2,3,4].map(function (val) {  
    return val * val;  
}); 

// squares will be equal to [1, 4, 9, 16]

Immutability — The original array will be unaffected. This has potential benefits in cases where the original array is still needed elsewhere. for loops can also be written so as not to update the original array, but it requires more code and updating our new array as part of our loop operation. On the other hand map() keeps this cleaner since you only have to work in one scope to still maintain immutability

Cleaner code — When doing identical things, map can almost always be written with less code than for. It can be clearly written on one line sometimes whereas for requires at least two or generally three with braces included. Also, scope isolation and a reduction in the number of variables you need alongside reduced size all make code objectively cleaner.

6. Rounding numbers

The toFixed() method converts a number rounding to a specified number of decimals.

var pi =3.1415;
pi = pi.toFixed(2);  // pi will be equal to 3.14

NOTE : toFixed() returns a string and not a number.

7. Use console.table

You can use console.table to show objects in tabular format:

table=[{state: "Texas"},{state: "New York"},{state: "Chicago"}]
console.table(table)

8. Avoid using try-catch inside a loop

The try-catch construct creates a new variable in the current scope at runtime each time the catch clause is executed where the caught exception object is assigned to a variable.

var object = ['foo', 'bar'], i;  
for (i = 0, len = object.length; i <len; i++) {  
    try {  
        // do something that throws an exception 
    }  
    catch (e) {   
        // handle exception  
    } 
}

to ⬇

var object = ['foo', 'bar'], i;  
try { 
    for (i = 0, len = object.length; i <len; i++) {  
        // do something that throws an exception 
    } 
} 
catch (e) {   
    // handle exception  
}

When an error occurs, the first one lets you continue the loop while the second one exits the loop. The first one is suited if an exception thrown by your code is not severe enough to halt your entire program.

9. Multiple condition checking

For multiple value matching, we can put all values in an array and use indexOf() or includes() method.

if (value === 1 || value === 'one' || value === 2 || value === 'two') { 

}

to ⬇

indexOf():

if ([1, 'one', 2, 'two'].indexOf(value) >= 0) { 

}

includes():

if ([1, 'one', 2, 'two'].includes(value)) { 

}

10. Double NOT bitwise operator (~~)

The double NOT bitwise operator is a substitute for Math.floor() method.

const floor = Math.floor(6.8); // 6

to ⬇

const floor = ~~6.8; // 6

The double NOT bitwise operator approach only works for 32-bit integers. So for any number higher than that, it is recommended to use Math.floor()

Conclusion

I know that there are many other tips, tricks, and best practices, so if you have any ones to add or if you have any feedback or corrections to the ones that I have shared, please feel free to add them in a comment below.

Also, You can never learn from just one article. Try Google concepts and read multiple articles, play around with the code by making projects and that's the only way you learn.

Here's my final tip — Don’t use a casual coding style. Enforce a standard

You never know what to expect while reading the code that has a random coding style. The same coding style across the entire team and the application codebase is a requirement. It’s a boost for code readability.

References

• https://modernweb.com/45-javascript-tips-tricks-practices/

• https://stackoverflow.com/questions/2485423/is-using-var-to-declare-variables-optional

• https://stackoverflow.com/questions/32844818/why-i-need-to-avoid-using-try-catch-finally-inside-a-loop

• http://rocha.la/JavaScript-bitwise-operators-in-practice

• Cover Photo by Juanjo Jaramillo on Unsplash

Python is a beautifully designed high-level interpreted programming language that provides us with many features.

This is a gentle guide to some of those Python features that you probably might not be aware of or didn't know about that specific use-case which you'll see later in this blog.

So in continuation of my previous article, here I am sharing another 5++ tips and tricks to help you write an elegant Python code in your competitive programming journey or in general as well! 😉

1. Lambda Function

A lambda function is a small anonymous function. You can use "lambda" expression to sort a nested array by the second element

>> arr_list = [[1, 4], [3, 3], [5, 7]]
>> sorted_list = sorted(arr_list , key=lambda x: x[1])
>> sorted_list 
[[3, 3], [1, 4], [5, 7]]

🔗Link To Tweet

2. Counter

Counter class is a special type of object data-set provided with the collections module in Python3.

Let's say we want an 𝐮𝐧𝐨𝐫𝐝𝐞𝐫𝐞𝐝 collection where elements are stored as 𝐝𝐢𝐜𝐭𝐢𝐨𝐧𝐚𝐫𝐲 𝐤𝐞𝐲𝐬 & their counts are stored as 𝐝𝐢𝐜𝐭𝐢𝐨𝐧𝐚𝐫𝐲 𝐯𝐚𝐥𝐮𝐞𝐬

import collections

>> A = collections.Counter([1, 1, 2, 2, 3, 3, 3, 3, 4, 5, 6, 7])
>> A
Counter({3: 4, 1: 2, 2: 2, 4: 1, 5: 1, 6: 1, 7: 1})
>> A.most_common(1)
[(3, 4)]
>>A.most_common(3)
[(3,4), (1, 2), (2, 2)]

🔗Link To Tweet

3. Merge 2 Dictionaries

Python 3.9 introduces Dictionary union (|)

It will return a new dictionary consisting of the left operand merged with the right operand. If a key appears in both operands, the last-seen value (i.e., from the right-hand operand) is selected.

>> d1 = {'a': 10, 'b': 5, 'c': 3}
>> d2 = {'d':6, 'c': 4, 'b': 8}
>> d1 | d2
{'a': 10, 'b': 8, 'c': 4, 'd': 6}
>> d2 | d1
{'d': 6, 'c': 3, 'b': 5, 'a': 10}

🔗Link To Tweet

4. Special Strip

Can you guess what will be the output of the following python code -

>>> "three cool features in Python".strip(" Python")

Well, strip() only remove the substring " Python" from the beginning or the end but the way it does it is quite peculiar

It removes the individual characters " ", "P", "y", "t", "h", "o", "n" instead of checking the whole "Python" as a word

Note that it even considers spaces while removing characters and It is also case sensitive

🔗Link To Tweet

5. [] + [] -> {}

Ever wanted to form a dictionary out of two lists??

In Python🐍 it is super easy

There's a zip() function that takes several iterable objects and returns a list of tuples

Each tuple groups the elements of the input objects by their positional index

>>> x = ['a', 'b', 'c']
>>> y = [1, 2, 3]
>>> dict(zip(x, y))
{'a': 1, 'b': 2, 'c': 3}

You can even use it when you want to transpose a 2-D matrix, like this ⬇

>>> mat = [[1, 2, 3], [4, 5, 6]]
>>> list(zip(*mat))
[(1, 4), (2, 5), (3, 6)]

🔗Link To Tweet

5++. Sliding Window

When I was doing competitive programming, I saw a lot of questions involving the sliding window problem

In C++ it was a little hectic, But in python, you can do it with just a few lines of code -

>>> from itertools import islice
>>> def slide(list_name, window_size):
...           z = [islice(list_name, i, None) for i in range(window_size)]
...           return zip(*z)

Here's the sample input and output response ⬇

>>> list(slide([1, 2, 3, 4, 5, 6, 7], 3))
[(1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6), (5, 6, 7)]

🔗Link To Tweet

Wrapping Up 🏁

So those were the 5++ cool quirks of python. I hope you liked them and learned something new.

Also if you enjoyed it, consider buying me a coffee ☕

(Seriously, I'd like some c̶o̶f̶f̶e̶e̶ support 😛)

Other articles you might like 😊

• Metrics to evaluate your Machine Learning algorithm 📊 - Evaluation Metrics for your Machine Learning algorithm that every Data Scientist must know.

• Why you can't name a file 'CON' in Windows ❓ - Learn why Windows doesn’t allow you to create files and folders named CON, PRN, AUX, and NUL.

• Five++ cool Python snippets that will blow your mind (Part - 1) 🤯 - Few clever yet useful Python tricks and tips that will surely make you think hard. This is Part-1 of this blog post series.

• Useful Resources To Learn Web Development & To Create Your Website 💻 - Starting with web development? Here are some resources that helped me and will be useful for you as well.

If you’re running a Windows machine, try it right now. Go to File Explorer -> create a new folder -> name it "Con"

Did it work? Now try with "PRN", "AUX" or "NUL". What did you get?

Probably this 👇

The system will not let you do it.

Con is an English word, and there are a few human names like Connor who goes by the nickname 'Con'. So if you’re trying to save a photo of one of them using their nickname on Windows, you will not able to.

In fact, the system won't let you name a file or folder CON, AUX, PRN, LST, COM0 to COM9, LPT0 to LPT9, or NUL.

But why Windows doesn’t allow this?

The reason for that goes back to MS-DOS, one of the oldest command lines from the 80s. In MS-DOS, all those words that are mentioned above are the names of device files.

They were a quick and clever way of letting programs on a computer interact with other hardware like printers, keyboards, and basically anything that you can plug in. Rather than every program having to learn to communicate with every possible device, that job was given to a "device driver".

This was done so that programmers didn’t have to worry about how every single type of hardware device like printers worked. Instead, they could reuse code that they already had for reading and writing files from and to the printer. They would just have to save whatever they wanted to print to this "device file".

This means that even when a new printer came out with new features in it and if the manufacturer would bundle a new driver with it, all the old programs would still work, in the same way as they always did, and the driver would deal with the dirty stuff.

Backward Compatibility

If you type "dir" in your windows terminal, you’d get a listing on the screen of whatever files were in the current directory or folder. Typing "dir", adding a redirect symbol(>) after that, and mentioning a file name at last, would then save the listing on that file instead of the listing going to the screen.

But if you tried to save it to a file called "LPT1", the system would spot that file name, and instead of going to the hard disk, your list will be sent to the device driver for the parallel port, the one that was labeled LPT1 on the back of your computer. The place where you have a printer plugged in. So even knowing anything about how the parallel port or printer worked, you could just print stuffs.

Windows 10 probably don't use parallel port now but the reason why these reserved words still exist till now is because of backward compatibility.

Windows 10 retains code from the days of Windows 3.1, and Windows 3.1 was created with MS-DOS as basic, so was Windows 95. With Windows NT this did not happen, but still, Microsoft wanted it to be compatible with Windows 95, and Windows 10 is still partly based on Windows NT. So those were the reserved words in MS-DOS in the 80s and they still continue to be reserved words in Windows in 2021

Almost no one may use a parallel port to connect a printer anymore, but Microsoft chooses to continue to support extremely old software and devices to this day.

Conclusion

In this article, we saw why you can't name a file or a folder "Con" in Windows.

In the early times of MS-DOS, computers were basically a black screen with text and nothing else. So everything worked by writing orders. For a program to interact with other things, such as a printer, a command had to be written using a text file or “device file” that would interact with a driver.

Depending on the device, it has a name reserved. For example, the parallel ports that printers used to connect to use LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. The serial ports has reserved COM from COM1 to COM9. "CON" represents the console etc.

To this day those device names are reserved so that you cannot create usable files or folders with them. If back then you tried to do something like creating a file called LPT1.txt, what you were going to do was send it to the printer or something like that and that’s why they were blocked.

Hope you learned something new today!

Thanks for reading :))

Other articles you might like 😊

• Metrics to evaluate your Machine Learning algorithm - Measure the quality of the statistical or machine learning model.

• Generating unique IDs in a Large scale Distributed environment - Learn how to create unique IDs for a distributed system at a large scale. Inspired from Twitter snowflake.

• Useful Resources To Learn Web Development & To Create Your Website - Resources I found helpful in my web development journey

But how do you check its performance and robustness? Simply building a predictive model is not enough. You have to create a model which gives high accuracy on out-of-sample data.

That's where evaluation metrics come into the picture. They are used to measure the quality of the statistical or machine learning model, and this article will introduce you to the most common and yet important metrics to evaluate your machine learning algorithms.

If you're someone who just started out in the field of Data Science and Machine Learning, this is for you. We will discuss the following terms:

• Confusion matrix
• Accuracy
• Precision
• Recall
• F1 score
• Mean Absolute Error
• R Squared Score
• Root Mean Squared Error
• ROC (Receiver Operating Characteristics) curve

So let’s dive into it to learn about them :

Confusion Matrix

A confusion matrix is one of the simplest concepts in the field of machine learning, and yet so important when it comes to statistical classification problems.

A confusion matrix is an N X N matrix, where N is the number of classes being predicted. It is usually used to describe the performance of a classification model based on the true values known for a data set.

In simple words, it is a summary of the prediction result made by the classification model.

Terms Associated with Confusion Matrix

True Positive (TP): True positive means their actual class value is 1 and the predicted value is also 1.

For example, The case where the woman is actually pregnant and the model also classifies that she is pregnant.

False Positive (FP): False positive means their actual class value is 0 and the predicted value is 1.

For example, The case where the woman is not pregnant and the model classifies that she is pregnant.

True Negative (TN): True negative means their actual class value is 0 and the predicted value is also 0.

For example, The case where the woman is actually not pregnant and the model also classifies that she is not pregnant.

False Negative (FN): False-negative means their actual class value is 1 and the predicted value is 0.

For example, The case where the woman is actually pregnant and the model also classifies that she is not pregnant.

The confusion matrix in itself is not a performance measure but all the performance metrics are dependent on the confusion matrix.

Accuracy

Accuracy is the most common metric that you can see everywhere when you're evaluating your model. It is simply defined as the number of correct predictions made by your model.

In simple terms, accuracy is the ratio of all the correct predictions to the total number of predictions.

It can be calculated using a confusion matrix.

Precision

Precision is defined as the total number of correctly classified positive examples by the total number of predicted positive examples.

In some cases, your classification model might classify based on the most frequent classes. Which in turn will bring a low accuracy because your model didn’t learn anything and just classified based on the top class.

Therefore, we need class-specific performance metrics to analyze. Precision is one of them.

It depicts how much the model is right when it says it is right.

Recall

A recall (also known as sensitivity) refers to the percentage of total relevant results correctly classified by the classification model.

It is the the number of positive samples returned by the custom-trained model.

The relation between recall and precision -

High recall, low precision:

This means, most of the positive examples are correctly recognized but there are a lot of false positives.

Low recall, high precision:

This means that we missed a lot of positive examples but those we predicted as positive are indeed positive.

F1 Score

Based on the use case of the classification model, the priority is given either to precision or recall, but in some classification models, we need both of these metrics to be combined as a single one.

F1-Score is a metric that combines both precision and recall and has an equal and relative contribution of both precision and recall.

It is the harmonic mean of precision and recall.

Remember that if your use case needs either recall or precision, one higher than the other then F1-score may not be the good metric for it.

Mean Absolute Error

Mean absolute error(MAE) is defined as the average of all errors that are calculated based on values predicted by your model.

It is intended to measure average model bias in a set of predictions, without considering their direction

R Squared Score

R squared is a measure of how close the data are to the fitted regression line. It is also known as the coefficient of determination.

It defines the degree to which the variance in the dependent variable (or target) can be explained by the independent variable (features).

For example, if the R-squared value for our predictive model is 0.8. This means that 80% of the variation in the dependent variable is explained by the independent variables.

Therefore we can say that the higher the r-squared value is, the better is the model.

SSRES: the Residual sum of squared errors

SSTOT: the total sum of squared errors

Root Mean Squared Error

RMSE is one of the most popular metrics used today for evaluating regression-based models. This is an important evaluation metric since it’s essential to find the average squared error between the predicted values.

RMSE measures the average magnitude of the error. It’s the square root of the average of squared differences between prediction and actual observation.

RMSE is highly affected by outlier values. Hence, make sure you’ve removed outliers from your data set prior to using this metric.

As compared to mean absolute error, RMSE gives higher weightage and punishes large errors

here 'N' is total observations.

Correlation between MAE and RMSE

Both of these metrics express the average error of the machine learning models. These two metrics can range from 0 to infinity and both of these metrics are negatively oriented scores, which means that a lower score defines better results.

ROC Curve

ROC curve stands for “Receiver Operating Characteristic" Curve.

The ROC curve is a graph showing the performance of a classification model at all its cut-off thresholds.

In simple words, the ROC Curve is the one that tells how much your model is capable of differentiating among the different classes.

ROC is a probability curve. It is a representation of the performance of your model in a graphical manner.

This curve represents:

1. True positive rate (recall/sensitivity)
2. False-positive rate (1- specificity): FPR = FP/ FP+TN

The curve separates the space into two areas, one for good and the other for poor performance levels.

Conclusion

In this article, you have seen some important metrics that are helpful for model evaluation and need to be understood by a data scientist or a machine learning engineer.

I hope you have found it useful. Also please feel free to suggest corrections and improvements.

And if you're looking for a good Data Science course for upskilling yourself, you might consider exploring the offerings of Odinschool.

Thanks for reading :))

Related articles you might like 😊

• Understanding Linear Regression - One of the most popular algorithms in machine learning that every data scientist should know.

• Having a go at common NLP tasks using TextBlob - Learn some of the basic operations to do in Natural language processing using TextBlob.

• Generating unique IDs in a Large-scale Distributed environment - Learn how to create unique IDs for a distributed system at a large scale. Inspired by Twitter snowflake.

• Useful Resources To Learn Web Development & To Create Your Website - Resources I found helpful in my web development journey

Recently at work, We were looking for a way to generate unique IDs across a distributed system that could also be used as the primary keys in the MySQL tables.

We knew in a single MySQL database we can simply use an auto-increment ID as the primary key, But this won’t work in a sharded MySQL database.

So I looked at various existing solutions for this and finally learned about Twitter Snowflake - a simple 64-bit unique ID generator.

Why don't you use UUID? 🤔

UUIDs are 128-bit hexadecimal numbers that are globally unique. The chances of the same UUID getting generated twice are negligible.

The problem with UUIDs is that they are very big in size and don’t index well. When your dataset increases, the index size increases as well and the query performance degrades.

Another problem with UUIDs is related to the user experience. Eventually, our users will be needed that unique identifiers. Imagine that a customer calls Customer Service and is asked to provide the identifier. Having to spell a complete UUID is not a pleasant experience.

Twitter snowflake ❄

Twitter snowflake is a dedicated service for generating 64-bit unique identifiers used in distributed computing for objects within Twitter such as Tweets, Direct Messages, Lists, etc.

These IDs are unique 64-bit unsigned integers, which are based on time. The full IDs are made up of the following components:

• Epoch timestamp in millisecond - 41 bits (gives us 69 years with respect to any custom epoch)

• Configured machine/node/shard Id - 10 bits (gives us up to total of 2¹⁰ i.e 1024 Ids)

• Sequence number - 12 bits (A local counter per machine that sets to zero after every 4096 values)

• The extra 1 reserved bit at the beginning which is set as 0 to make the overall number as positive.

Since these use the timestamp as the first component, therefore, they are time sortable as well. Another benefit is its High Availability.

By default, 64-bit unsigned integers (long) will generate an Id whose length is 19, but sometimes it may be too long, our use case needed an Id whose length should not be greater than 10.

This article will share a simplified version of the unique ID generator that will work for any use-case of generating unique IDs in a distributed environment based on the concepts outlined in the Twitter snowflake service.

Coding time ⌚

In our case, the full ID will be composed of a 20-bit timestamp, 5-bit worker number, and 6-bit sequence number.

The remaining 1-bit is the signed bit and it is always set to 0 to make the final value positive.

Our microservices can use this Random number generator to generate IDs independently. This is efficient and fits in the size of a int (4 Bytes or 32 bits).

Here is the complete code in Java (Inspired by Twitter snowflake, code credits) -

Step 1 - We initialize the number of bits that each component will require :

public class Snowflake {

    // Sign bit, Unused (always set to 0)
    private static final int UNUSED_BITS = 1; 

    private static final int EPOCH_BITS = 20;
    private static final int NODE_ID_BITS = 5;
    private static final int SEQUENCE_BITS = 6;

    private static final int maxNodeId = (int)(Math.pow(2, NODE_ID_BITS) - 1);
    private static final int maxSequence = (int)(Math.pow(2, SEQUENCE_BITS) - 1);

    // Custom Epoch (Fri, 21 May 2021 03:00:20 GMT)
    private static final int DEFAULT_CUSTOM_EPOCH = 1621566020;

    private volatile int lastTimestamp = -1;
    private volatile int sequence = 0;

    // Class Constructor
    public Snowflake() {
        this.nodeId = createNodeId();
        this.customEpoch = DEFAULT_CUSTOM_EPOCH;
    }

Here, we are taking custom epoch as of Fri, 21 May 2021 03:00:20 GMT.

EPOCH_BITS will be 20 bits and is filled with a current timestamp in seconds (You can also use millisecond if there is a possibility of multiple numbers of requests per second).

NODE_ID_BITS will be 5 bits and is filled using the Mac address.

SEQUENCE_BITS will be 6 bits and will act as a local counter which will start from 0, goes till 63, and then resets back to 0.

Step 2 - Creating a synchronized function to generate the IDs :

 public synchronized int nextId() {
        int currentTimestamp = (int) (Instant.now().getEpochSecond() - customEpoch);

        if(currentTimestamp < lastTimestamp) {
            throw new IllegalStateException("Invalid System Clock!");
        }

        lastTimestamp = currentTimestamp;

        return currentTimestamp << (NODE_ID_BITS + SEQUENCE_BITS) | (nodeId << SEQUENCE_BITS) | sequence;
    }

Wait, why are we doing these Left Shifts & Logical OR operations?

This is because Integer is represented by 32 bits and initially all are set to 0. To fill these bits we have to take each component separately, so first we took the epoch timestamp and shift it to 5 + 6 i.e 11 bits to left. Doing this has filled the first 21 bits with the first component (remember the first bit is always set to zero to make the overall number positive). The remaining 11 bits are still 0 and hence again we repeat the same thing with logical OR & the other two components as well thereby filling all the 32 bits and forming the complete number.

Step 3 - Utility function to generate the node id using the system’s MAC address:

private int createNodeId() {
        int nodeId;
        try {
            StringBuilder sb = new StringBuilder();
            Enumeration<NetworkInterface> networkInterfaces = NetworkInterface.getNetworkInterfaces();
            while (networkInterfaces.hasMoreElements()) {
                NetworkInterface networkInterface = networkInterfaces.nextElement();
                byte[] mac = networkInterface.getHardwareAddress();
                if (Objects.nonNull(mac))
                    for(byte macPort: mac)
                        sb.append(String.format("%02X", macPort));
            }
            nodeId = sb.toString().hashCode();
        } catch (Exception ex) {
            nodeId = (new SecureRandom().nextInt());
        }
        nodeId = nodeId & maxNodeId;
        return nodeId;
    }
}

How it works? 💡

Let’s now understand its working with an example -

Let’s say it’s Sun, 23 May 2021 00:00:00 GMT right now. The epoch timestamp for this particular time is 1621728000.

First of all, we adjust our timestamp with respect to the custom epoch-

currentTimestamp = 1621728000- 1621566020 = 161980(Adjust for custom epoch)

So to start our ID, the first 20 bits of the ID (after the signed bit) will be filled with the epoch timestamp. Let's this value with a left-shift :

id = currentTimestamp << (NODE_ID_BITS + SEQUENCE_BITS )

Next, we take the configured node ID/shard ID and fill the next 10 bits with that

id = id | nodeId << SEQUENCE_BITS

Finally, we take the next value of our auto-increment sequence and fill out the remaining 6 bits -

id = id | sequence // 6149376

That gives us our final ID 🎉

And that’s it! Primary keys that are unique across our application!

Summary 📊

This article showed you a simple solution of how to generate a snowflake id whose length is >=7 and <=10.

By the way, you can adjust the bit count of the 3 components to adapt to your work.

NOTE : We should keep the generator as a singleton, it means that we should only create the single instance of SequenceGenerator per node. If not, it may generate some duplicate Ids.

Not only did twitter used it, Discord also uses snowflakes, with their epoch set to the first second of the year 2015.

Instagram uses a modified version of the format, with 41 bits for a timestamp, 13 bits for a shard ID, and 10 bits for a sequence number.

I hope this will help you! Thanks for reading :))

Other Articles You Might Like 🙂

• Useful Resources To Learn Web Development & To Create Your Website - Resources I found helpful in my web development journey.

• Chrome extensions I use to enhance my GITHUB experience - Here are 7 extensions I use to improve my Github experience.

• The Most Famous Coding Interview Question - Learn about what's the most asked coding interview question and how you can approach it in the best possible way.

• Welcome to the world of "NFTs" - Learn about what are NFTs and Why are they suddenly becoming the next big thing.

• What is Blockchain Technology? Learn by creating one - Want to know how blockchain works? Learn here by creating one.

Software Development is one of those fields that is fast-paced, and always in flux. It’s what makes the field so challenging, interesting, and fun all at the same time.

But that also means that you will constantly keep coming across the stuff you don’t know. And even the stuff that you use to think you know but actually you don’t, which is why a lot of us face the situation like the imposter syndrome and the Dunning Kruger effect (a lot!).

Imposter Syndrome

For those unfamiliar with the term imposter syndrome, It is when a person starts to doubt their own capabilities & accomplishments and believes that they are a fraud and have succeeded only through luck and soon people will realize they don't know what they are talking about.

It is when someone is being too hard on themselves and not giving themselves the credit for what they deserve and achieved. Instead, the achieving individuals believe that they have been only lucky in their achievements and they haven't done anything major to achieve it.

Is it good or bad?

I think the imposter syndrome is both good and bad.

It's bad because this can become a cause of anxiety and unhappiness in someone's life, who by all means should be proud of their work and accomplishments.

Furthermore, I think why so many software developers face imposter syndrome is because -

• Measuring all the work

Whatever solution you come up with, it’s very easy to measure the “efficiency” of that solutions. And no one would feel proud if another developer came in and improved the code by 500% in a just few days where you have spent perhaps weeks or months.

This leads to the thinking that they're not qualified enough and all that they have achieved is solely through luck.

• Global Competition

From online coding contests to interviewing to raising PRs for open-source to full-fledged industry-level product development. The software industry is very big and growing very fast, so the competition is inevitable and global – it’s not limited by physical/geographic distance.

And that is just another reason to not feel comfortable at all times.

• Forgetting the difficulty of things

In the process of comparing, or working alongside highly capable developers, it's very easy to lose track of the high quality of work that’s going around. But I think it's OK to feel mediocre as long as you know you’re learning, growing, and contributing back to the community.

At the same time why I feel it is good is because -

Here are some reasons why:

• It Indicates you’re pushing yourself

• It acts as a signal that you’re gaining expertise

Dunning-Kruger effect

The Dunning-Kruger effect is a cognitive bias stating the relationship between an individual’s perceived knowledge of the subject matter and its ACTUAL knowledge.

In 1999, Dunning and Kruger conducted a study: Unskilled and Unaware of It: How difficulties in Recognizing One's Incompetence Lead to Inflated Self-Assessments

The result of their study found that those who are incompetent tend to overestimate their abilities, while those who are competent underestimate them

If you observe the above graph you’ll see that beginners or less experienced devs are mostly under the impression that they are familiar with all that exists in their field and they're Pro.

In the simpler way of interpreting the graph, we can say, The inexperienced do not know what they do not know.

But WHY does it happen?

It happens because the experienced developers, or those on the track to becoming experienced, know how vast the field of software development is. They know what they don’t understand.

Can you see how this can cause an imposter syndrome to come up?

By knowing this effect in addition to being aware of imposter syndrome, we can handle imposter syndrome better.

Managing Imposter Syndrome

In general, I believe the following tips help to handle imposter syndrome while understanding the Dunning Kruger Effect:

• Make a list of your accomplishments Track your achievements. All of them. It doesn't matter how big or small it is. When something good happens, just make a note of it with a description of the effort you took to achieve it.

• Understand the bell curve The bell curves depict that most of us are indeed average, so by learning a bit more each day or week you gradually move towards the right part of the graph. And that's what we should aim for as well.

• Peer Pairing Making a “peer pair” with someone who is going through the same situation as what you’re going through. It can be helpful as you'll be able to discuss both your success and failures with them. There’s a lot of fellow beginners out there in the developer community and everyone is a newbie in something. So, you can easily find someone with whom you can share your learnings.

Apart from these suggestions, I also believe the most important part in preventing imposter syndrome comes from focusing on becoming a life-long learner. Take some time to yourself each day to read about new developments in your field, train on a technology you haven't yet used.

Pick something and practice it, regularly.

The Dunning Kruger Effect shows that the more you know, the more you feel like you don't - accept and prosper from this. If you ever feel like you don't know anything, just remember It actually means progress, It means growth, It means you still have areas where you can grow and learn

Thanks for reading :)

Other Articles You Might Like 🙂

• Useful Resources To Learn Web Development & To Create Your Website - Resources I found helpful in my web development journey.

• Chrome extensions I use to enhance my GITHUB experience - Here are 7 extensions I use to improve my Github experience.

• The Most Famous Coding Interview Question - What's the most asked coding interview question? And how to approach it.

• Welcome to the world of "NFTs" - What are NFTs? And why are they suddenly becoming the next big thing?

• What is Blockchain Technology? Learn by creating one - What is a blockchain? How do they work? Learn by creating one.

Let's have a look at some of them 🔽

1. Octotree

It enhances GitHub code review and exploration with features like:

• IDE-like code tree
• Folder and file search
• Issues and Pull Request bookmarking
• Support for private repositories

I like this extension very much! It lets you explore the files and folders of a repository with a tree like structure.

⭐ Github Repo | Chrome Extension

2. Open in VS Code

It Opens links to source code for any github repository in VSCode. Also works with Gitlab.

💡 Tip: In case you want to open the whole repository in GitHub directly with VSCode view rather than a single file, Just add 1s after github. Example: github.com/user/somerepository -> github1s.com/user/somerepository.

⭐ Github Repo | Chrome Extension

3. Refined GitHub

It simplifies the GitHub interface and adds useful features too, for instance:

• Adds reaction avatars showing who reacted to a comment
• Adds one-click merge conflict fixers
• Adds a button to revert all the changes to a file in a PR

⭐ Github Repo | Chrome Extension

4. Do-not-merge-WIP

Disables "Merge pull request" button while ANY of the following is true:

• Title contains "[wip]" (case insensitive)
• Any tasks remain incomplete
• Any commit messages remain prefixed with "squash!" or "fixup!"

⭐ Github Repo | Chrome Extension

5. Enhanced GitHub

It provides useful features on top of GitHub Website:

• Displays repo size
• Displays each file size for every active branch
• Show download link for each file
• Copy file's contents directly to Clipboard
• Download file while viewing its contents.

⭐ Github Repo | Chrome Extension

6. Githunt

It lets you hunt the most starred projects on GitHub on any date. Very useful when you want to explore the most trending repo of the week/month/year.

⭐ Github Repo | Chrome Extension

7. CodeCopy

It lets you copy the text from any file on github to your clipboard. Apart from Github it also works on :

• Stack Overflow
• MDN
• Medium
• npm
• Gist

⭐ Github Repo | Chrome Extension

Wrapping Up 👋

So with this, we come to the end of the list! In case you know another extension that I haven't included but you find it useful, please feel free to share those in the comments below😀

I hope you have enjoyed this article and found it helpful.

Thanks For Reading :)

"Enjoyed what you read??" You can show some support💙 by Buying me some coffee☕