<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://spsarolkar.github.io/feed.xml" rel="self" type="application/atom+xml"/><link href="https://spsarolkar.github.io/" rel="alternate" type="text/html" hreflang="en"/><updated>2026-02-17T17:23:02+00:00</updated><id>https://spsarolkar.github.io/feed.xml</id><title type="html">blank</title><subtitle>This is a profile page of Sunil Sarolkar </subtitle><entry><title type="html">Building GCP ML Engineer Mock Exam App with GitHub Pages</title><link href="https://spsarolkar.github.io/blog/2025/Building-GCP-ML-Engineer-MockExam-app-with-Github-pages/" rel="alternate" type="text/html" title="Building GCP ML Engineer Mock Exam App with GitHub Pages"/><published>2025-07-05T00:00:00+00:00</published><updated>2025-07-05T00:00:00+00:00</updated><id>https://spsarolkar.github.io/blog/2025/Building-GCP-ML-Engineer-MockExam-app-with-Github-pages</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2025/Building-GCP-ML-Engineer-MockExam-app-with-Github-pages/"><![CDATA[<p>If you’re preparing for the <strong>Google Cloud Certified - Professional Machine Learning Engineer</strong> exam and want a structured, interactive way to practice, I’ve built a free mock exam tool — and you can too! Here’s how I created a professional, responsive <strong>mock test web app</strong> hosted on GitHub Pages using Jekyll, YAML, and TailwindCSS.</p> <hr/> <h2 id="-why-i-built-this">💡 Why I Built This</h2> <p>As I was preparing for the GCP ML Engineer exam, I noticed a lack of high-quality, interactive mock exams — especially ones that are open source and customizable. So I decided to build one myself with a few goals in mind:</p> <ul> <li>✅ Structured questions based on the <strong>official exam guide</strong></li> <li>✅ Filterable by <strong>tags and sections</strong></li> <li>✅ Timer-based <strong>exam simulation</strong></li> <li>✅ Immediate <strong>feedback and score</strong></li> <li>✅ Review incorrect answers and <strong>download as PDF</strong></li> <li>✅ Hosted freely using <strong>GitHub Pages</strong></li> </ul> <hr/> <h2 id="-tech-stack">🔨 Tech Stack</h2> <ul> <li><strong>Jekyll</strong> (for static site generation)</li> <li><strong>TailwindCSS</strong> (for styling)</li> <li><strong>JavaScript</strong> (no frameworks)</li> <li><strong>YAML</strong> (for storing questions by section)</li> <li><strong>GitHub Pages</strong> (for free hosting)</li> </ul> <hr/> <h2 id="-features">🧠 Features</h2> <table> <thead> <tr> <th>Feature</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>🧪 Full &amp; Quick Exam Modes</td> <td>Simulate a 120-minute full exam or a 30-minute practice sprint</td> </tr> <tr> <td>🏷️ Tag Filtering</td> <td>Filter questions by topics like Vertex AI, Feature Engineering, etc.</td> </tr> <tr> <td>📋 Section-Based Sampling</td> <td>Questions weighted by domain (%), as in the official exam guide</td> </tr> <tr> <td>⏰ Timer &amp; Auto-Submit</td> <td>Countdown timer with automatic submission on timeout</td> </tr> <tr> <td>📊 Summary View</td> <td>Full answer breakdown with highlights for correct/incorrect choices</td> </tr> <tr> <td>⬇️ PDF Download</td> <td>Print-friendly summary to save results</td> </tr> <tr> <td>🧩 Review Incorrect Only</td> <td>Focus your prep on the questions you missed</td> </tr> </tbody> </table> <hr/> <h2 id="️-a-visual-overview">🖼️ A Visual Overview</h2> <h3 id="welcome-page">Welcome page</h3> <p><img src="assets/blog/GCP-ML-MockExamapp/WelcomePage.png" alt="GCP ML Mock Exam Welcome Page" title="GCP ML Mock Exam Welcome Page"/></p> <h3 id="selecting-exam-mode">Selecting Exam Mode</h3> <p><img src="assets/blog/GCP-ML-MockExamapp/SelectingExamMode.png" alt="GCP ML Mock Exam Selecting Exam Mode" title="Selecting Mock Exam Mode"/></p> <hr/> <h2 id="-try-the-app">🚀 Try the App</h2> <p>👉 <strong><a href="https://spsarolkar.github.io/ml-cert-mock-exam/">Launch Mock Exam App</a></strong><br/> No login. No cookies. Just practice.</p> <hr/> <h2 id="-how-it-works">📁 How It Works</h2> <h3 id="folder-structure">Folder Structure</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ml-cert-mock-exam/
├── index.html            # Landing page
├── mock-test.html        # Main app
├── _data/sections/       # YAML files for question banks
├── README.md
├── CONTRIBUTING.md
└── thumbnail.png
</code></pre></div></div> <h3 id="yaml-example-section1yml">YAML Example (<code class="language-plaintext highlighter-rouge">section1.yml</code>)</h3> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">question</span><span class="pi">:</span> <span class="s2">"</span><span class="s">When</span><span class="nv"> </span><span class="s">leveraging</span><span class="nv"> </span><span class="s">BigQuery</span><span class="nv"> </span><span class="s">ML,</span><span class="nv"> </span><span class="s">which</span><span class="nv"> </span><span class="s">consideration</span><span class="nv"> </span><span class="s">focuses</span><span class="nv"> </span><span class="s">on</span><span class="nv"> </span><span class="s">defining</span><span class="nv"> </span><span class="s">the</span><span class="nv"> </span><span class="s">problem</span><span class="nv"> </span><span class="s">structure</span><span class="nv"> </span><span class="s">and</span><span class="nv"> </span><span class="s">desired</span><span class="nv"> </span><span class="s">outcome</span><span class="nv"> </span><span class="s">for</span><span class="nv"> </span><span class="s">the</span><span class="nv"> </span><span class="s">model?"</span>
  <span class="na">options</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">Building the appropriate BigQuery ML model based on the business problem</span>
    <span class="pi">-</span> <span class="s">Optimizing hardware accelerators</span>
    <span class="pi">-</span> <span class="s">Integrating with external data sources</span>
    <span class="pi">-</span> <span class="s">Monitoring model performance in real-time</span>
  <span class="na">answer</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">tags</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">BigQuery ML</span>
    <span class="pi">-</span> <span class="s">Business Problem</span>
</code></pre></div></div> <hr/> <h2 id="-want-to-build-one-yourself">💻 Want to Build One Yourself?</h2> <p>The entire project is open source.</p> <p>👉 <a href="https://github.com/spsarolkar/ml-cert-mock-exam">View on GitHub</a></p> <p>You can fork it, extend it with your own questions, or use it for other certifications (like AWS, Azure, or Kubernetes).</p> <hr/> <h2 id="-disclaimer">📢 Disclaimer</h2> <p>This project is <strong>not affiliated with or endorsed by Google</strong>.<br/> All questions are <strong>original</strong>, inspired by the <a href="https://services.google.com/fh/files/misc/professional_machine_learning_engineer_exam_guide_english.pdf">official exam guide</a> and intended for learning purposes only.</p> <hr/> <h2 id="-final-thoughts">🙌 Final Thoughts</h2> <p>I hope this tool helps you <strong>gain confidence</strong> for your exam day. If you find it useful, feel free to ⭐️ star the repo or suggest improvements via a pull request.</p> <p>Good luck on your ML journey — and may your models always converge!</p>]]></content><author><name>Sunil Sarolkar</name></author><category term="GCP"/><category term="Machine Learning"/><category term="Certification"/><category term="Mock Exam"/><category term="Jekyll"/><category term="GitHub Pages"/><summary type="html"><![CDATA[If you’re preparing for the Google Cloud Certified - Professional Machine Learning Engineer exam and want a structured, interactive way to practice, I’ve built a free mock exam tool — and you can too! Here’s how I created a professional, responsive mock test web app hosted on GitHub Pages using Jekyll, YAML, and TailwindCSS.]]></summary></entry><entry><title type="html">StructFormer: Transformer-based Structured Data Adjustment Generator</title><link href="https://spsarolkar.github.io/blog/2025/Structformer-Structural-Data-Transformer/" rel="alternate" type="text/html" title="StructFormer: Transformer-based Structured Data Adjustment Generator"/><published>2025-04-19T12:40:00+00:00</published><updated>2025-04-19T12:40:00+00:00</updated><id>https://spsarolkar.github.io/blog/2025/Structformer-Structural-Data-Transformer</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2025/Structformer-Structural-Data-Transformer/"><![CDATA[<p>I recently completed a project called <strong>StructFormer</strong>, a Transformer-based model that generates <strong>SQL adjustment statements</strong> from structured error records. The model is particularly useful in enterprise data processing pipelines where large-scale reconciliation or error adjustment tasks are automated.</p> <h3 id="-problem-statement">🔍 Problem Statement</h3> <p>In financial or trading systems, data errors such as “Incorrect Account Type” or “Missing Quantity” are common and typically resolved by writing SQL adjustments. My goal was to create a <strong>sequence-to-sequence model</strong> that learns to generate such adjustments from natural language-like structured inputs.</p> <p>Example:</p> <p><strong>Input:</strong></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TradeID=50874 AccountID=ACC1003 ErrorType=Negative Amount
</code></pre></div></div> <p><strong>Expected Output:</strong></p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">UPDATE</span> <span class="n">Trades</span> <span class="k">SET</span> <span class="n">Amount</span><span class="o">=</span><span class="mi">831</span><span class="p">.</span><span class="mi">05</span> <span class="k">WHERE</span> <span class="n">TradeID</span><span class="o">=</span><span class="mi">50874</span><span class="p">;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">AdjustmentLog</span><span class="p">(</span><span class="n">ErrorID</span><span class="p">,</span> <span class="n">AdjustedBy</span><span class="p">)</span> <span class="k">VALUES</span><span class="p">(</span><span class="s1">'ERR5827'</span><span class="p">,</span> <span class="s1">'User1'</span><span class="p">);</span>
</code></pre></div></div> <h3 id="-model-architecture">🧠 Model Architecture</h3> <ul> <li>Built using Keras (TensorFlow backend)</li> <li>Transformer Encoder-Decoder architecture</li> <li>Positional embeddings from Keras Hub</li> <li>Trained with SentencePiece tokenizer (custom-trained on domain corpus)</li> <li>Custom decoder inference using greedy decoding</li> </ul> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@keras.saving.register_keras_serializable</span><span class="p">(</span><span class="n">package</span><span class="o">=</span><span class="sh">"</span><span class="s">transformerEncoder</span><span class="sh">"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">TransformerEncoder</span><span class="p">(</span><span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="n">Layer</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">,</span><span class="n">hidden_dim</span><span class="p">,</span><span class="n">intermediate_dim</span><span class="p">,</span><span class="n">num_heads</span><span class="p">,</span><span class="n">dropout_rate</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span><span class="n">name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
        <span class="nf">super</span><span class="p">().</span><span class="nf">__init__</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
        <span class="n">key_dim</span> <span class="o">=</span> <span class="n">hidden_dim</span>
        <span class="n">self</span><span class="p">.</span><span class="n">intermediate_dim</span> <span class="o">=</span> <span class="n">intermediate_dim</span>
        <span class="n">self</span><span class="p">.</span><span class="n">num_heads</span> <span class="o">=</span> <span class="n">num_heads</span>
        <span class="n">self</span><span class="p">.</span><span class="n">dropout_rate</span> <span class="o">=</span> <span class="n">dropout_rate</span>
        <span class="n">self</span><span class="p">.</span><span class="n">self_attention</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">MultiHeadAttention</span><span class="p">(</span><span class="n">num_heads</span><span class="p">,</span><span class="n">key_dim</span><span class="p">)</span>
        <span class="n">self</span><span class="p">.</span><span class="n">self_attn_layer_norm</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">LayerNormalization</span><span class="p">()</span>
        <span class="n">self</span><span class="p">.</span><span class="n">ff_1</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">Dense</span><span class="p">(</span><span class="n">intermediate_dim</span><span class="p">,</span><span class="n">activation</span><span class="o">=</span><span class="sh">"</span><span class="s">relu</span><span class="sh">"</span><span class="p">)</span>
        <span class="n">self</span><span class="p">.</span><span class="n">ff_2</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">Dense</span><span class="p">(</span><span class="n">hidden_dim</span><span class="p">)</span>
        <span class="n">self</span><span class="p">.</span><span class="n">ff_layer_norm</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">LayerNormalization</span><span class="p">()</span>
        <span class="c1"># self.dropout_layer=keras.layers.Dropout(dropout_rate)
</span>
    <span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">self</span><span class="p">,</span><span class="n">source</span><span class="p">,</span><span class="n">source_mask</span><span class="p">):</span>
      <span class="n">residual</span> <span class="o">=</span> <span class="n">x</span> <span class="o">=</span> <span class="n">source</span>
      <span class="n">mask</span> <span class="o">=</span> <span class="n">source_mask</span><span class="p">[:,</span><span class="bp">None</span><span class="p">,:]</span>
      <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">self_attention</span><span class="p">(</span><span class="n">query</span> <span class="o">=</span> <span class="n">x</span><span class="p">,</span><span class="n">value</span> <span class="o">=</span> <span class="n">x</span><span class="p">,</span><span class="n">key</span> <span class="o">=</span> <span class="n">x</span><span class="p">)</span><span class="c1">#, attention_mask=tf.cast(mask, tf.float32)) # This is specifically required for M1 Mac
</span>      <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">residual</span>
      <span class="n">x</span> <span class="o">=</span><span class="n">self</span><span class="p">.</span><span class="nf">self_attn_layer_norm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
      <span class="n">residual</span> <span class="o">=</span> <span class="n">x</span>
      <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">ff_1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
      <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">ff_2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
      <span class="n">x</span> <span class="o">=</span> <span class="n">x</span><span class="o">+</span><span class="n">residual</span>
      <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">ff_layer_norm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
      <span class="k">return</span> <span class="n">x</span>
</code></pre></div></div> <p>The decoder is similarly structured, using both causal and cross attention.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@keras.saving.register_keras_serializable</span><span class="p">(</span><span class="n">package</span><span class="o">=</span><span class="sh">"</span><span class="s">transformerDecoder</span><span class="sh">"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">TransformerDecoder</span><span class="p">(</span><span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="n">Layer</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">hidden_dim</span><span class="p">,</span> <span class="n">intermediate_dim</span><span class="p">,</span> <span class="n">num_heads</span><span class="p">,</span><span class="n">name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
        <span class="nf">super</span><span class="p">().</span><span class="nf">__init__</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
        <span class="n">key_dim</span> <span class="o">=</span> <span class="n">hidden_dim</span> <span class="o">//</span> <span class="n">num_heads</span>
        <span class="n">self</span><span class="p">.</span><span class="n">self_attention</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">MultiHeadAttention</span><span class="p">(</span><span class="n">num_heads</span><span class="p">,</span> <span class="n">key_dim</span><span class="p">)</span>
        <span class="n">self</span><span class="p">.</span><span class="n">self_attention_layernorm</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">LayerNormalization</span><span class="p">()</span>
        <span class="n">self</span><span class="p">.</span><span class="n">cross_attention</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">MultiHeadAttention</span><span class="p">(</span><span class="n">num_heads</span><span class="p">,</span> <span class="n">key_dim</span><span class="p">)</span>
        <span class="n">self</span><span class="p">.</span><span class="n">cross_attention_layernorm</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">LayerNormalization</span><span class="p">()</span>
        <span class="n">self</span><span class="p">.</span><span class="n">feed_forward_1</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">Dense</span><span class="p">(</span><span class="n">intermediate_dim</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="sh">"</span><span class="s">relu</span><span class="sh">"</span><span class="p">)</span>
        <span class="n">self</span><span class="p">.</span><span class="n">feed_forward_2</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">Dense</span><span class="p">(</span><span class="n">hidden_dim</span><span class="p">)</span>
        <span class="n">self</span><span class="p">.</span><span class="n">feed_forward_layernorm</span> <span class="o">=</span> <span class="n">keras</span><span class="p">.</span><span class="n">layers</span><span class="p">.</span><span class="nc">LayerNormalization</span><span class="p">()</span>

    <span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="n">source</span><span class="p">,</span> <span class="n">source_mask</span><span class="p">):</span>
        <span class="n">residual</span> <span class="o">=</span> <span class="n">x</span> <span class="o">=</span> <span class="n">target</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">self_attention</span><span class="p">(</span><span class="n">query</span><span class="o">=</span><span class="n">x</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">x</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="n">x</span><span class="p">,</span> <span class="n">use_causal_mask</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">residual</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">self_attention_layernorm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
        <span class="n">residual</span> <span class="o">=</span> <span class="n">x</span>
        <span class="n">mask</span> <span class="o">=</span> <span class="n">source_mask</span><span class="p">[:,</span> <span class="bp">None</span><span class="p">,</span> <span class="p">:]</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">cross_attention</span><span class="p">(</span>
            <span class="n">query</span><span class="o">=</span><span class="n">x</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="n">source</span><span class="c1">#, attention_mask=tf.cast(mask, tf.float32) # This is specifically required for M1 Mac
</span>        <span class="p">)</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">residual</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">cross_attention_layernorm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
        <span class="n">residual</span> <span class="o">=</span> <span class="n">x</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">feed_forward_1</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">feed_forward_2</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">residual</span>
        <span class="n">x</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="nf">feed_forward_layernorm</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">x</span>
</code></pre></div></div> <h3 id="️-technologies-used">🛠️ Technologies Used</h3> <ul> <li>TensorFlow / Keras 3</li> <li>SentencePiece tokenizer</li> <li>NumPy &amp; Pandas for data preprocessing</li> <li>Jupyter Notebooks for development</li> <li>FastAPI + React (planned deployment)</li> <li>Optional: Spark for scalable pre-tokenization</li> </ul> <h3 id="-results">🔍 Results</h3> <p>After extensive training, the model reached <strong>~99% validation accuracy</strong> using windowing-based training, token-level padding, and beam search refinement. Below are some sample predictions:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Input: TradeID=29216 AccountID=ACC1003 ErrorType=Incorrect Account Type
Expected: UPDATE Accounts SET AccountType='Savings' WHERE AccountID='ACC1003'...
Predicted: UPDATE Accounts SET AccountType='Checking' WHERE AccountID='ACC1003'...
</code></pre></div></div> <p>Even when predictions differ, they are syntactically valid and often semantically close.</p> <h3 id="-what-makes-this-unique">🧹 What Makes This Unique?</h3> <ul> <li>Works well with <strong>real-world structured data</strong></li> <li>Can adapt to new error types via <strong>fine-tuning</strong></li> <li>Supports custom lookup tables (e.g., currencies, account types)</li> <li>Tokenization designed to handle numerical and domain-specific vocabulary</li> </ul> <h3 id="-github-repository">📆 GitHub Repository</h3> <p>Check out the full codebase, training script, and inference pipeline here:</p> <p>👉 <a href="https://github.com/spsarolkar/StructFormer">https://github.com/spsarolkar/StructFormer</a></p> <h3 id="-whats-next">📈 What’s Next?</h3> <ul> <li>Add REST API using FastAPI</li> <li>Integrate model with a Streamlit/React dashboard</li> <li>Enable multi-record batching and validation UI</li> </ul> <hr/> <p>This was a fascinating experiment, and I plan to evolve it into a plug-and-play solution for <strong>automated structured data correction</strong> in enterprise applications.</p> <p>Feel free to fork, contribute, or try it on your own datasets! 🚀</p>]]></content><author><name></name></author><category term="Transformer"/><category term="Structured"/><category term="Data"/><category term="NLP"/><summary type="html"><![CDATA[I recently completed a project called StructFormer, a Transformer-based model that generates SQL adjustment statements from structured error records. The model is particularly useful in enterprise data processing pipelines where large-scale reconciliation or error adjustment tasks are automated.]]></summary></entry><entry><title type="html">Vehicle Telemetry using Raspberry Pi with Kafka-based Anomaly Detection</title><link href="https://spsarolkar.github.io/blog/2025/Vehicle-Telemetry-For-Failure-Detection/" rel="alternate" type="text/html" title="Vehicle Telemetry using Raspberry Pi with Kafka-based Anomaly Detection"/><published>2025-04-16T14:40:00+00:00</published><updated>2025-04-16T14:40:00+00:00</updated><id>https://spsarolkar.github.io/blog/2025/Vehicle-Telemetry-For-Failure-Detection</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2025/Vehicle-Telemetry-For-Failure-Detection/"><![CDATA[<p>In 2019, I built a hobby project to collect real-time vehicle parameters from the <strong>OBD-II port</strong> using an <strong>ELM327 Bluetooth adapter</strong> and a <strong>Raspberry Pi</strong>. This setup allowed me to display <strong>vehicle speed</strong> and <strong>RPM</strong> on a custom instrument console. As a demonstration, the Raspberry Pi connected to the ELM327 via Bluetooth and parsed live vehicle data directly from the CAN bus.</p> <p><img src="(/assets/projects/VehicleTelemetry/OBD2_Vehicle_detection.png)" alt="Architecture Diagram"/></p> <p>The hardware flow looked like this:</p> <ul> <li><strong>Vehicle</strong> → ELM327 / CAN Bus</li> <li><strong>ELM327</strong> → Bluetooth → <strong>Raspberry Pi</strong></li> <li><strong>Raspberry Pi</strong> → Instrument Console (Python-based UI)</li> </ul> <p>I have now planned to enhance the project with real-time streaming and anomaly detection:</p> <ul> <li>The Raspberry Pi connects to a <strong>Kafka broker</strong> using a <strong>4G modem</strong>.</li> <li>Vehicle telemetry data such as speed, RPM, coolant temperature, and throttle position is pushed to Kafka topics.</li> <li>These streams are processed using <strong>Apache Spark</strong>, and anomalies are detected with a trained <strong>ML model</strong>.</li> </ul> <h2 id="-implemented-components">✅ Implemented Components:</h2> <ul> <li>ELM327 Bluetooth integration</li> <li>Real-time data collection on Raspberry Pi</li> <li>Custom instrument display for speed and RPM</li> </ul> <h2 id="-planned-enhancements">🔜 Planned Enhancements:</h2> <ul> <li>Kafka streaming pipeline with 4G modem</li> <li>Apache Spark for telemetry processing</li> <li>ML-based failure detection using vehicle health indicators</li> </ul> <h2 id="demo--source-code">Demo &amp; Source Code</h2> <p>You can check out the working video demo and the source code in the GitHub repository:</p> <iframe width="560" height="315" src="https://www.youtube.com/embed/X3aA26rKwIs?si=Wro3g8q26U2lR8Gr" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen=""></iframe> <h2 id="github-repo">Github Repo</h2> <p>💻 <strong><a href="https://github.com/spsarolkar/Tesla/tree/master">GitHub Repo</a></strong></p> <h2 id="tech-stack">Tech Stack</h2> <ul> <li>Raspberry Pi 3B+</li> <li>Python 3</li> <li>ELM327 OBD-II Adapter</li> <li>Apache Kafka (planned)</li> <li>Apache Spark MLlib (planned)</li> <li>4G USB Modem</li> </ul> <h2 id="final-thoughts">Final Thoughts</h2> <p>This project connects the physical and digital world by capturing automotive telemetry and enabling intelligent, ML-based diagnostics. It’s an exciting intersection of <strong>IoT</strong>, <strong>machine learning</strong>, and <strong>real-time streaming</strong>—a great example of applying data science beyond traditional dashboards.</p> <p>If you’re working on similar projects or have suggestions for failure prediction features, feel free to connect or fork the repo!</p>]]></content><author><name></name></author><category term="vehicle"/><category term="telemetry"/><category term="anomaly"/><category term="detection"/><category term="raspberrypi"/><category term="kafka"/><summary type="html"><![CDATA[In 2019, I built a hobby project to collect real-time vehicle parameters from the OBD-II port using an ELM327 Bluetooth adapter and a Raspberry Pi. This setup allowed me to display vehicle speed and RPM on a custom instrument console. As a demonstration, the Raspberry Pi connected to the ELM327 via Bluetooth and parsed live vehicle data directly from the CAN bus.]]></summary></entry><entry><title type="html">Redis High Availability Solution using Redis Sentinel and docker</title><link href="https://spsarolkar.github.io/blog/2019/Configuring-Redis-High-Availability-Sentinel-Configuration/" rel="alternate" type="text/html" title="Redis High Availability Solution using Redis Sentinel and docker"/><published>2019-12-29T12:15:00+00:00</published><updated>2019-12-29T12:15:00+00:00</updated><id>https://spsarolkar.github.io/blog/2019/Configuring-Redis-High-Availability-Sentinel-Configuration</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2019/Configuring-Redis-High-Availability-Sentinel-Configuration/"><![CDATA[<p>In this post we will configure the Redis high availability server using kubernetes. Below will be our final configuration for redis cluster.</p> <p><img src="/assets/blog/RedisHighAvailability/Redis-High-Availability-server-block-diagram.png" alt="Redis-High-Availability-server-block-diagram"/></p> <p>For each block present above we will have seperate docker container. So we have below list of images</p> <ol> <li>master-sentinel: This will be single copy of container which will run resid master node and redis sentinel node</li> <li>replica-sentinel: This will be multiple(two in our case above) copy of container which will run redis replica node and redis sentinel node</li> </ol> <h6 id="creating-docker-image-for-master-sentinel"><strong>Creating Docker image for master-sentinel</strong></h6> <p>To configure master-sentinel image we need to two redis instances running, one with master configuration and one with sentinel configuration. So there will be two seperate configurations for both thes instances.</p> <ol> <li> <p><strong>redis-master.conf</strong></p> <p>This configuration file will be used for starting Redis as a master node. As we are running the redis inside docker container it will aquire the local ip address assigned by Kubernetes, redis replica and redis sentinel processes on different container will need docker pod ip address to connect to this master node. So we need special configuration –cluster-announce-ip, –cluster-announce-port and –sentinel announce-ip, –sentinel announce-port that will be provided as a environment variable by Kubernetes deployment configuration. As we need master to listen connections from other docker machines we need to comment the line <code class="language-plaintext highlighter-rouge">bind 127.0.0.1</code> in default configurations of redis.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># bind 127.0.0.1
</code></pre></div> </div> <p>Also protected mode should be turned off by commenting <code class="language-plaintext highlighter-rouge">protected-mode yes</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># protected-mode yes
</code></pre></div> </div> <p>We also need configuration added from command line so that process outside container will be able to connect to master node.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>    --cluster-announce-ip $MY_POD_IP
    --cluster-announce-port $MASTER_ANNOUNCE_PORT
</code></pre></div> </div> </li> <li> <p><strong>redis-sentinel.conf</strong> We will be configurting ht sentinel configuration from command line while starting the sentinel process from start script using below arguments</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--sentinel announce-ip $MY_POD_IP --sentinel announce-port $SENTINEL_ANNOUNCE_PORT

--sentinel monitor mymaster $MY_POD_IP $MASTER_ANNOUNCE_PORT 2

--sentinel down-after-milliseconds mymaster 30000

--sentinel parallel-syncs mymaster 1

--sentinel failover-timeout mymaster 180000
</code></pre></div> </div> <p>Please note that we have hardcoded the master group <code class="language-plaintext highlighter-rouge">mymaster</code>. We need to use this while connecting from client.</p> </li> <li> <p><strong>start.sh</strong></p> <p>We need start script that will start both instances of redis(master and sentinel) with configuration discussed above.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>redis-server /usr/local/etc/redis/redis.conf <span class="nt">--cluster-announce-ip</span> <span class="nv">$MY_POD_IP</span> <span class="nt">--cluster-announce-port</span> <span class="nv">$MASTER_ANNOUNCE_PORT</span>
</code></pre></div> </div> <p>Note that we are using environment variable for <code class="language-plaintext highlighter-rouge">--cluster-announce-ip</code> and <code class="language-plaintext highlighter-rouge">--cluster-announce-port</code> this needs to be a part of kubernetes deployment cofiguration</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>redis-server /usr/local/etc/redis/sentinel.conf <span class="nt">--sentinel</span> announce-ip <span class="nv">$MY_POD_IP</span> <span class="nt">--sentinel</span> announce-port <span class="nv">$SENTINEL_ANNOUNCE_PORT</span> <span class="nt">--sentinel</span> monitor mymaster <span class="nv">$MY_POD_IP</span> <span class="nv">$MASTER_ANNOUNCE_PORT</span> 2 <span class="nt">--sentinel</span> down-after-milliseconds mymaster 30000 <span class="nt">--sentinel</span> parallel-syncs mymaster 1 <span class="nt">--sentinel</span> failover-timeout mymaster 180000
</code></pre></div> </div> </li> </ol> <h6 id="creating-docker-image-for-replica-sentinel"><strong>Creating docker image for replica-sentinel</strong></h6> <p>For creating image for replica-sentinel we need docker replica and docker sentinel running in same container and both should be connecting to redis master instance inside master-sentinel image. We will have configurations for redis-replica and redis-sentinel for the same.</p> <ol> <li> <p><strong>redis-replicator.conf</strong></p> <p>We need redis-master to be able to connect to redis-replicator so we need below configuration commented</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># bind 127.0.0.1
</code></pre></div> </div> <p>Also we need to turn off protected mode by commenting below line</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># protected-mode yes
</code></pre></div> </div> <p>We will also use below configuration from command line arguments</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--replica-announce-ip $MY_POD_IP
--replicaof $REDIS_MASTER_SERVICE_HOST $REDIS_MASTER_SERVICE_PORT

</code></pre></div> </div> <p>We need to pass $MY_POD_IP, $REDIS_MASTER_SERVICE_HOST and $REDIS_MASTER_SERVICE_PORT environment varialbles from kubernetes deployment configurations</p> </li> <li> <p><strong>redis-sentinel.conf</strong></p> <p>Sentinel configuration will be similar to the sentinel configuration inside master-sentinel, only different is instead of connecting to master node on the same machine it will read the master hostname from environment variable,</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--sentinel monitor mymaster $REDIS_MASTER_SERVICE_HOST $REDIS_MASTER_SERVICE_PORT 2
</code></pre></div> </div> <p>Rest of the configuration is same as sentinel in master-sentinel image.</p> </li> <li> <p><strong>start.sh</strong></p> <p>Finally we will have start.sh starting the replica and sentinel processes</p> <p>Starting replica instance</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>redis-server /usr/local/etc/redis/redis-replicator.conf --replica-announce-ip $MY_POD_IP --replicaof $REDIS_MASTER_SERVICE_HOST $REDIS_MASTER_SERVICE_PORT &amp;&gt; /var/log/redis-replica.log &amp;
</code></pre></div> </div> <p>Starting sentinel instance</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>redis-server /usr/local/etc/redis/sentinel.conf --sentinel announce-ip $MY_POD_IP --sentinel announce-port $SENTINEL_ANNOUNCE_PORT --sentinel monitor mymaster $REDIS_MASTER_SERVICE_HOST $REDIS_MASTER_SERVICE_PORT 2 --sentinel down-after-milliseconds mymaster 30000 --sentinel parallel-syncs mymaster 1 --sentinel failover-timeout mymaster 180000
</code></pre></div> </div> </li> </ol> <h6 id="kubernetes-configuration-for-master-sentinel-and-replica-sentinel"><strong>Kubernetes configuration for master-sentinel and replica-sentinel</strong></h6> <p>For deploying the redis cluster on Kubernetes we need to create deployment and services as below <img src="/assets/blog/RedisHighAvailability/Kubernetes-deployment-redis-high-availability.png" alt="Kubernetes-deployment-redis-high-availability"/></p> <p><strong>master-sentinel deployment</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-primary</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
  <span class="na">labels</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis-primary</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">redis-primary</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">redis-primary</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">redis</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">spsarolkar/master-sentinel:1.0</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">100m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">100Mi</span>
          <span class="na">env</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">MY_POD_IP</span>
              <span class="na">valueFrom</span><span class="pi">:</span>
                <span class="na">fieldRef</span><span class="pi">:</span>
                  <span class="na">fieldPath</span><span class="pi">:</span> <span class="s">status.podIP</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">MASTER_ANNOUNCE_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">6379"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SENTINEL_ANNOUNCE_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">26379"</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">6379</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">redis-master</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">26379</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">redis-sentinel</span>
</code></pre></div></div> <p><strong>master-sentinel service</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-primary</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">ports</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">6379</span>
      <span class="na">targetPort</span><span class="pi">:</span> <span class="m">6379</span>
      <span class="na">name</span><span class="pi">:</span> <span class="s">redis-master</span>
    <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">26379</span>
      <span class="na">targetPort</span><span class="pi">:</span> <span class="m">26379</span>
      <span class="na">name</span><span class="pi">:</span> <span class="s">redis-sentinel</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis-primary</span>
</code></pre></div></div> <p><strong>replica-sentinel deployment</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-secondary</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
  <span class="na">labels</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis-secondary</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">redis-secondary</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">2</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">redis-secondary</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">replica</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">spsarolkar/replica-sentinel:1.0</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">100m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">100Mi</span>
          <span class="na">env</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_MASTER_SERVICE_HOST</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s">redis-primary</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_MASTER_SERVICE_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">6379"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SENTINEL_ANNOUNCE_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">26379"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">MY_POD_IP</span>
              <span class="na">valueFrom</span><span class="pi">:</span>
                <span class="na">fieldRef</span><span class="pi">:</span>
                  <span class="na">fieldPath</span><span class="pi">:</span> <span class="s">status.podIP</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">26379</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">redis-sentinel</span>
</code></pre></div></div> <p><strong>replica-sentinel service</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-secondary</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">ports</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">26379</span>
      <span class="na">targetPort</span><span class="pi">:</span> <span class="m">26379</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis-secondary</span>
</code></pre></div></div> <h6 id="connecting-to-redis-from-client-application"><strong>Connecting to redis from client application</strong></h6> <p>To use this high availability cluster we need client capable of connecting to redis sentinel. In the example application we will use spring based configuration for connecting to redis sentinel.</p> <p>I have created simple application using spring that has one deamon service and one front end service. The application block diagram is as below</p> <p><img src="/assets/blog/RedisHighAvailability/Redis-High-Availability-server-block-diagram.png" alt="FortuneTeller-cowsay-block-diagram"/></p> <ol> <li> <p><strong>Fortune Teller UI</strong></p> <p>This is simple spring boot application configured to use open id from google for authorization purpose. We have configured the HTTP session to connect to redis sentinel using below configuration</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Bean</span>
<span class="kd">public</span> <span class="nc">LettuceConnectionFactory</span> <span class="nf">connectionFactory</span><span class="o">()</span> <span class="o">{</span>
    <span class="nc">RedisSentinelConfiguration</span> <span class="n">config</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RedisSentinelConfiguration</span><span class="o">()</span>
            <span class="o">.</span><span class="na">master</span><span class="o">(</span><span class="n">appConfig</span><span class="o">.</span><span class="na">getRedisSentinelMasterGroup</span><span class="o">())</span>
            <span class="o">.</span><span class="na">sentinel</span><span class="o">(</span><span class="n">appConfig</span><span class="o">.</span><span class="na">getRedisMasterSentinelName</span><span class="o">(),</span><span class="n">appConfig</span><span class="o">.</span><span class="na">getRedisMasterSentinelPort</span><span class="o">())</span>
            <span class="o">.</span><span class="na">sentinel</span><span class="o">(</span><span class="n">appConfig</span><span class="o">.</span><span class="na">getRedisReplicaSentinelName</span><span class="o">(),</span><span class="n">appConfig</span><span class="o">.</span><span class="na">getRedisReplicaSentinelPort</span><span class="o">());</span>
    <span class="nc">LettuceConnectionFactory</span> <span class="n">connection</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">LettuceConnectionFactory</span><span class="o">(</span><span class="n">config</span><span class="o">);</span>
    <span class="k">return</span> <span class="n">connection</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div> </div> <p><strong>fortune-teller deployment</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">fortune-teller</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
  <span class="na">labels</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">fortune-teller</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">3</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">fortune-teller</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">fortune-teller</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">fortune-teller</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">spsarolkar/fortune-teller:2.0</span>
          <span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">IfNotPresent</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">8080</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">limits</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">500m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">700Mi</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">100m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">300Mi</span>
          <span class="na">env</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">COWSAY_SERVER_NAME</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">cowsay-deamon"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">COWSAY_SERVER_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">8000"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_MASTER_NAME</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">redis-primary"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">MASTER_SENTINEL_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">26379"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_REPLICA_NAME</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">redis-secondary"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REPLICA_SENTINEL_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">26379"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_SENTINEL_MASTER_GROUP</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mymaster"</span>
</code></pre></div> </div> <p><strong>fortune-teller-service</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">fortune-teller-service</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">LoadBalancer</span>
  <span class="na">ports</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
      <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
      <span class="na">port</span><span class="pi">:</span> <span class="m">80</span>
      <span class="na">targetPort</span><span class="pi">:</span> <span class="m">8080</span>
      <span class="na">nodePort</span><span class="pi">:</span> <span class="m">31737</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">fortune-teller</span>
</code></pre></div> </div> </li> <li> <p><strong>Cowsay deamon</strong></p> <p>This is rest API built using Django python framework. Front end will call rest service exposed by this deamon and will produce the fortune messages.</p> <p><strong>cowsay-deployment</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay-deployment</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
  <span class="na">labels</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay-deployment</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">2</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay-deployment</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay-deployment</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay-deployment</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">spsarolkar/cowsay-deamon:1.0</span>
          <span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">IfNotPresent</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">8000</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">limits</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">200m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">300Mi</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">200m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">300Mi</span>
</code></pre></div> </div> <p><strong>cowsay-service</strong></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay-deamon</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">ports</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
      <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
      <span class="na">port</span><span class="pi">:</span> <span class="m">8000</span>
      <span class="na">targetPort</span><span class="pi">:</span> <span class="s">http</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay-deployment</span>
</code></pre></div> </div> </li> </ol> <p>I have deployed example application on minikube and is accesible from link <a href="http://gcp-kube-demo.sunilsarolkar.com/">http://gcp-kube-demo.sunilsarolkar.com/</a>.</p> <p>Source code available in Github repository <a href="https://github.com/spsarolkar/kube-redis-sample-setup">https://github.com/spsarolkar/kube-redis-sample-setup</a>.</p>]]></content><author><name></name></author><category term="Redis"/><category term="High-Availability"/><category term="Sentinel"/><category term="kubernetes"/><summary type="html"><![CDATA[In this post we will configure the Redis high availability server using kubernetes. Below will be our final configuration for redis cluster.]]></summary></entry><entry><title type="html">Polybar does not support the bridge interface for network tracking</title><link href="https://spsarolkar.github.io/blog/2018/Plybar-does-not-support-bridge-interface/" rel="alternate" type="text/html" title="Polybar does not support the bridge interface for network tracking"/><published>2018-11-25T12:32:15+00:00</published><updated>2018-11-25T12:32:15+00:00</updated><id>https://spsarolkar.github.io/blog/2018/Plybar-does-not-support-bridge-interface</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2018/Plybar-does-not-support-bridge-interface/"><![CDATA[<p><a href="https://github.com/jaagr/polybar">Polybar</a> is one of the most popular status bar for tiled window manager. Like most of the statusbars it has network module to track upload and download traffic for interface configured in the configuration file.</p> <p>In my case I had bridge interface configured so that I could add qemu tap interface to bridge interface so that guest virtual machine will be able to reach internet. I have i3 window manager installed on my gentoo linux with stock i3bar, I wanted to change my taskbar to Polybar which has better look and easy configration.</p> <p>When I configured the bridge interface Polybar</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>module/eth]
<span class="nb">type</span> <span class="o">=</span> internal/network
interface <span class="o">=</span> br0
interval <span class="o">=</span> 3.0

format-connected-underline <span class="o">=</span> <span class="c">#55aa55</span>
format-connected-prefix <span class="o">=</span> <span class="s2">""</span>
format-connected-prefix-foreground <span class="o">=</span> <span class="k">${</span><span class="nv">colors</span><span class="p">.foreground-alt</span><span class="k">}</span>
label-connected <span class="o">=</span> %local_ip%
</code></pre></div></div> <p>In the configuration above I just wanted to display the public ip address of the bridge interface.</p> <p>When I started the polybar it started showing error message indicating its not able to connect to my bridge interface <code class="language-plaintext highlighter-rouge">br0</code></p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>warn: module/eth: Failed to query interface <span class="s1">'br0'</span>
</code></pre></div></div> <p>After few google searches I found the existing issue reported by someone with respect to bridge interface. So I took up the challenge to dig up the code to figure out the problem.</p> <p>Polybar has network modules which makes use of <code class="language-plaintext highlighter-rouge">ioctl</code> to determine the link speed of the physical interface which of course fails for bridge network being the virtual rather than physical.</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ioctl</span><span class="p">(</span><span class="o">*</span><span class="n">m_socketfd</span><span class="p">,</span> <span class="n">SIOCETHTOOL</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">request</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span>
</code></pre></div></div> <p>So even though it could compute the network ip address and upload download speed, it fails on the call to <code class="language-plaintext highlighter-rouge">ioctl</code>. On close observation of the code I could see there is already a special case for <code class="language-plaintext highlighter-rouge">tun</code>/<code class="language-plaintext highlighter-rouge">tap</code> interface which is also a virtual device but code was not designed to handle bridge interface.</p> <p>So I updated the code to handle this special case for bridge interface along with <code class="language-plaintext highlighter-rouge">tun</code> <code class="language-plaintext highlighter-rouge">tap</code>.</p> <p>Below function ealier only handled the <code class="language-plaintext highlighter-rouge">tun</code> <code class="language-plaintext highlighter-rouge">tap</code> interfaces, I have added additional if block to mark the new variable <code class="language-plaintext highlighter-rouge">m_bridge</code> to indicate the bridge interface.</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="n">network</span><span class="o">::</span><span class="n">check_tuntap_or_bridge</span><span class="p">()</span> <span class="p">{</span>
    <span class="k">struct</span> <span class="n">ethtool_drvinfo</span> <span class="n">driver</span> <span class="p">{};</span>
    <span class="k">struct</span> <span class="n">ifreq</span> <span class="n">request</span> <span class="p">{};</span>

    <span class="n">driver</span><span class="p">.</span><span class="n">cmd</span> <span class="o">=</span> <span class="n">ETHTOOL_GDRVINFO</span><span class="p">;</span>

    <span class="n">memset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">request</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">request</span><span class="p">));</span>

    <span class="cm">/*
     * Only copy array size minus one bytes over to ensure there is a
     * terminating NUL byte (which is guaranteed by memset)
     */</span>
    <span class="n">strncpy</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">ifr_name</span><span class="p">,</span> <span class="n">m_interface</span><span class="p">.</span><span class="n">c_str</span><span class="p">(),</span> <span class="n">IFNAMSIZ</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>

    <span class="n">request</span><span class="p">.</span><span class="n">ifr_data</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">char</span><span class="o">*&gt;</span><span class="p">(</span><span class="o">&amp;</span><span class="n">driver</span><span class="p">);</span>

    <span class="k">if</span> <span class="p">(</span><span class="n">ioctl</span><span class="p">(</span><span class="o">*</span><span class="n">m_socketfd</span><span class="p">,</span> <span class="n">SIOCETHTOOL</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">request</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">return</span><span class="p">;</span>
    <span class="p">}</span>

    <span class="c1">// Check if it's a TUN/TAP device</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">strncmp</span><span class="p">(</span><span class="n">driver</span><span class="p">.</span><span class="n">bus_info</span><span class="p">,</span> <span class="s">"tun"</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">m_tuntap</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">strncmp</span><span class="p">(</span><span class="n">driver</span><span class="p">.</span><span class="n">bus_info</span><span class="p">,</span> <span class="s">"tap"</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">m_tuntap</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
      <span class="n">m_tuntap</span> <span class="o">=</span> <span class="nb">false</span><span class="p">;</span>
    <span class="p">}</span>

    <span class="k">if</span> <span class="p">(</span><span class="n">strncmp</span><span class="p">(</span><span class="n">driver</span><span class="p">.</span><span class="n">driver</span><span class="p">,</span> <span class="s">"bridge"</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">m_bridge</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span>
    <span class="p">}</span>

  <span class="p">}</span>
</code></pre></div></div> <p>Once we have interface with <code class="language-plaintext highlighter-rouge">m_bridge</code> flag set we can make use of this flag to skip the link speed calculation</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">f</span><span class="p">(</span><span class="o">!</span><span class="n">m_bridge</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// If bridge network then link speed cannot be computed TODO: Identify the physical network in bridge and compute the link speed</span>
	  <span class="k">struct</span> <span class="n">ifreq</span> <span class="n">request</span> <span class="p">{};</span>
	  <span class="k">struct</span> <span class="n">ethtool_cmd</span> <span class="n">data</span> <span class="p">{};</span>

	  <span class="n">memset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">request</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">request</span><span class="p">));</span>
	  <span class="n">strncpy</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">ifr_name</span><span class="p">,</span> <span class="n">m_interface</span><span class="p">.</span><span class="n">c_str</span><span class="p">(),</span> <span class="n">IFNAMSIZ</span> <span class="o">-</span> <span class="mi">1</span><span class="p">);</span>
	  <span class="n">data</span><span class="p">.</span><span class="n">cmd</span> <span class="o">=</span> <span class="n">ETHTOOL_GSET</span><span class="p">;</span>
	  <span class="n">request</span><span class="p">.</span><span class="n">ifr_data</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">char</span><span class="o">*&gt;</span><span class="p">(</span><span class="o">&amp;</span><span class="n">data</span><span class="p">);</span>

	  <span class="k">if</span> <span class="p">(</span><span class="n">ioctl</span><span class="p">(</span><span class="o">*</span><span class="n">m_socketfd</span><span class="p">,</span> <span class="n">SIOCETHTOOL</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">request</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
	    <span class="k">return</span> <span class="nb">false</span><span class="p">;</span>
	  <span class="p">}</span>

	  <span class="n">m_linkspeed</span> <span class="o">=</span> <span class="n">data</span><span class="p">.</span><span class="n">speed</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div> <p>Once above code in place it stated the polybar without any error.</p> <p>I have created the <a href="https://github.com/jaagr/polybar/pull/1528">pull request</a> for the change, once approve it will be available in the main build.</p>]]></content><author><name></name></author><category term="Polybar"/><category term="bridge"/><summary type="html"><![CDATA[Polybar is one of the most popular status bar for tiled window manager. Like most of the statusbars it has network module to track upload and download traffic for interface configured in the configuration file.]]></summary></entry><entry><title type="html">Google cloud kubernetes example</title><link href="https://spsarolkar.github.io/blog/2018/Google-cloud-kubernetes-example/" rel="alternate" type="text/html" title="Google cloud kubernetes example"/><published>2018-10-06T12:32:15+00:00</published><updated>2018-10-06T12:32:15+00:00</updated><id>https://spsarolkar.github.io/blog/2018/Google-cloud-kubernetes-example</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2018/Google-cloud-kubernetes-example/"><![CDATA[<p>We will convert the microservice we built in <a href="https://spsarolkar.github.io/docker/microservices/2018/09/30/Building-Microservices-Using-Docker.html">earlier post</a> and we will port it in google cloud platform using kubernetes. All the source code is available on <a href="https://github.com/spsarolkar/google-cloud-kubernetes-example">Github</a></p> <p>Our microservice consists of following parts</p> <ul> <li>Redis database for session management across multiple UI instances</li> <li>Backend rest service which generates the message text written using Django</li> <li>Fortune Teller front end written in Spring with Oauth2 authorization using Google account</li> </ul> <h4 id="redis-database-master-and-slave-configuration">Redis database master and slave configuration</h4> <p>We will create the deployment of master and slave using existing images available on Google registry.</p> <h5 id="redis-master">Redis master</h5> <p>We will configure the single instance deployment of redis master, below is the configuration for the same</p> <p>redis-master.yml</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-master</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">redis</span>
      <span class="na">role</span><span class="pi">:</span> <span class="s">primary</span>
      <span class="na">tier</span><span class="pi">:</span> <span class="s">backend</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">redis</span>
        <span class="na">role</span><span class="pi">:</span> <span class="s">primary</span>
        <span class="na">tier</span><span class="pi">:</span> <span class="s">backend</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">redis</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">gcr.io/google_containers/redis:e2e</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">100m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">100Mi</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">6379</span>
</code></pre></div></div> <p>On Kubernetes we will create the deployment of redis master using below command</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply <span class="nt">-f</span> redis-master.yml
</code></pre></div></div> <p>Redis master will be exposed on external port <code class="language-plaintext highlighter-rouge">port: 6379</code> with below service configuration</p> <p>redis-master-service.yml</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-master</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">ports</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">6379</span>
      <span class="na">targetPort</span><span class="pi">:</span> <span class="m">6379</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis</span>
    <span class="na">role</span><span class="pi">:</span> <span class="s">primary</span>
    <span class="na">tier</span><span class="pi">:</span> <span class="s">backend</span>
</code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply <span class="nt">-f</span> redis-master-service.yml
</code></pre></div></div> <h5 id="redis-slave">Redis slave</h5> <p>We will need redis-slave which would be doing actual work behind the scenes, in case of slave we will have 2 replicas. Redis slave will register itself on master using environment variable <code class="language-plaintext highlighter-rouge">REDIS_MASTER_SERVICE_HOST</code></p> <p>redis-slave.yml</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-replica</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">redis</span>
      <span class="na">role</span><span class="pi">:</span> <span class="s">replica</span>
      <span class="na">tier</span><span class="pi">:</span> <span class="s">backend</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">2</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">redis</span>
        <span class="na">role</span><span class="pi">:</span> <span class="s">replica</span>
        <span class="na">tier</span><span class="pi">:</span> <span class="s">backend</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">replica</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">gcr.io/google-samples/gb-redisslave@sha256:57730a481f97b3321138161ba2c8c9ca3b32df32ce9180e4029e6940446800ec</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">100m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">100Mi</span>
          <span class="na">env</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GET_HOSTS_FROM</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s">env</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_MASTER_SERVICE_HOST</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s">redis-master</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">6379</span>
</code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply -f redis-slave.yml
</code></pre></div></div> <p>redis-slave-service.yml</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-replica</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
  <span class="na">labels</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis</span>
    <span class="na">role</span><span class="pi">:</span> <span class="s">replica</span>
    <span class="na">tier</span><span class="pi">:</span> <span class="s">backend</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">ports</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">6379</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis</span>
    <span class="na">role</span><span class="pi">:</span> <span class="s">replica</span>
    <span class="na">tier</span><span class="pi">:</span> <span class="s">backend</span>
</code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply -f redis-slave-service.yml
</code></pre></div></div> <h4 id="cowsay-rest-service">Cowsay rest service</h4> <p>We will have cowsay rest service exposing its rest endpoint which will in turn be used in the front end.</p> <p>cowsay.yml</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">2</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">asia.gcr.io/fortune-teller-215315/cowsay@sha256:7f8e43af9af53cf59265836814eb21b9260b478f4ae5e4e8244ef322c20307ef</span>
          <span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">IfNotPresent</span>
          <span class="na">livenessProbe</span><span class="pi">:</span>
            <span class="na">failureThreshold</span><span class="pi">:</span> <span class="m">3</span>
            <span class="na">httpGet</span><span class="pi">:</span>
              <span class="na">path</span><span class="pi">:</span> <span class="s">/</span>
              <span class="na">port</span><span class="pi">:</span> <span class="m">8000</span>
              <span class="na">scheme</span><span class="pi">:</span> <span class="s">HTTP</span>
            <span class="na">initialDelaySeconds</span><span class="pi">:</span> <span class="m">3</span>
            <span class="na">periodSeconds</span><span class="pi">:</span> <span class="m">3</span>
            <span class="na">successThreshold</span><span class="pi">:</span> <span class="m">1</span>
            <span class="na">timeoutSeconds</span><span class="pi">:</span> <span class="m">1</span>
          <span class="na">readinessProbe</span><span class="pi">:</span>
            <span class="na">failureThreshold</span><span class="pi">:</span> <span class="m">1</span>
            <span class="na">httpGet</span><span class="pi">:</span>
              <span class="na">path</span><span class="pi">:</span> <span class="s">/</span>
              <span class="na">port</span><span class="pi">:</span> <span class="m">8000</span>
              <span class="na">scheme</span><span class="pi">:</span> <span class="s">HTTP</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">8000</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">limits</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">200m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">300Mi</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">200m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">300Mi</span>
</code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply <span class="nt">-f</span> cowsay.yml
</code></pre></div></div> <p>cowsay-service.yml</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
  <span class="na">labels</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">ports</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
      <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span>
      <span class="na">port</span><span class="pi">:</span> <span class="m">8000</span>
      <span class="na">targetPort</span><span class="pi">:</span> <span class="s">http</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay</span>
</code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply <span class="nt">-f</span> cowsay-service.yml
</code></pre></div></div> <h4 id="fortune-teller-front-end">Fortune Teller front end</h4> <p>We need deployment and service configuration for our application. Deployment file contains the configuration with respect to docker image, number of replicas of the runtime pods and other details like CPU /memory requirement. Our deployment file looks like below</p> <p>cowsay-ui.yml</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay-ui</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">web</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">2</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay-ui</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">cowsay-ui</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cowsay-ui</span>
          <span class="na">image</span><span class="pi">:</span> <span class="s">asia.gcr.io/fortune-teller-215315/fortune-teller-ui@sha256:416792ad7f37af47c1f775d6289ab63744df90fe7afd3cd3fcb5c8dc7b96270c</span>
          <span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">IfNotPresent</span>
          <span class="na">ports</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">8080</span>
              <span class="na">name</span><span class="pi">:</span> <span class="s">http</span>
          <span class="na">resources</span><span class="pi">:</span>
            <span class="na">limits</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">50m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">500Mi</span>
            <span class="na">requests</span><span class="pi">:</span>
              <span class="na">cpu</span><span class="pi">:</span> <span class="s">50m</span>
              <span class="na">memory</span><span class="pi">:</span> <span class="s">500Mi</span>
          <span class="na">env</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">COWSAY_SERVER_NAME</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">cowsay"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">COWSAY_SERVER_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">8000"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_SERVER_NAME</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">redis-master"</span>
            <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_SERVER_PORT</span>
              <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">6379"</span>
</code></pre></div></div> <p>As you can see above configuration contains the docker image details. In this case we are referring to image from Google registry but images from dockerhub should also work fine. We also mentioned the <code class="language-plaintext highlighter-rouge">REDIS</code> and backend <code class="language-plaintext highlighter-rouge">cowsay</code> service details.</p> <p>We can create the deployment using below command on Kubernetes cluster</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply -f cowsay-ui.yml
</code></pre></div></div> <p>Once deployment is finished our front end container will be started. We need expose this to external world using <code class="language-plaintext highlighter-rouge">LoadBalancer</code> service as below</p> <p>cowsay-ui-service.yml</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apiVersion: v1
kind: Service
metadata:
  name: cowsay-ui
  namespace: web
  labels:
    app: cowsay-ui
spec:
  type: LoadBalancer
  ports:
     - name: http
       protocol: TCP
       port: 80
       targetPort: 8080
  selector:
      app: cowsay-ui
</code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply -f cowsay-ui-service.yml
</code></pre></div></div> <p>To check all the services running on kubernetes use command <code class="language-plaintext highlighter-rouge">kubectl get service</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kubectl get service
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
cowsay          ClusterIP      10.39.241.184   &lt;none&gt;           8000/TCP       3d
cowsay-ui       LoadBalancer   10.39.253.234   35.200.227.184   80:32099/TCP   3d
redis-master    ClusterIP      10.39.250.176   &lt;none&gt;           6379/TCP       4d
redis-replica   ClusterIP      10.39.240.117   &lt;none&gt;           6379/TCP       4d
</code></pre></div></div> <p>This will show you all the services running and their respective IP addresses. Also it will show external ip address for container exposed as a <code class="language-plaintext highlighter-rouge">LoadBalancer</code> service</p> <p>I have currently hosted the application on <a href="http://gcp-kube-demo.sunilsarolkar.com">http://gcp-kube-demo.sunilsarolkar.com</a>. I will keep it alive until I can afford it.</p>]]></content><author><name></name></author><category term="docker"/><category term="microservices"/><category term="google-cloud"/><category term="kubernetes"/><summary type="html"><![CDATA[We will convert the microservice we built in earlier post and we will port it in google cloud platform using kubernetes. All the source code is available on Github]]></summary></entry><entry><title type="html">Building microservices using docker swarm with Oauth2</title><link href="https://spsarolkar.github.io/blog/2018/Building-Microservices-Using-Docker/" rel="alternate" type="text/html" title="Building microservices using docker swarm with Oauth2"/><published>2018-09-30T12:32:15+00:00</published><updated>2018-09-30T12:32:15+00:00</updated><id>https://spsarolkar.github.io/blog/2018/Building-Microservices-Using-Docker</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2018/Building-Microservices-Using-Docker/"><![CDATA[<p>Today we will learn how to build simple multi tier microservice using Oauth2 authorization with Redis session management. Our front end will be build using Spring Boot which will use backend python service for fetching the fortune message to be displayed. The block diagram for the application is as below. All the example code is uploaded to github at <a href="https://github.com/spsarolkar/docker-swarm-example">Github</a></p> <div class="mxgraph" style="max-width:100%;border:1px solid transparent;" data-mxgraph="{&quot;highlight&quot;:&quot;#0000ff&quot;,&quot;nav&quot;:true,&quot;resize&quot;:true,&quot;toolbar&quot;:&quot;zoom layers lightbox&quot;,&quot;edit&quot;:&quot;_blank&quot;,&quot;xml&quot;:&quot;&lt;mxfile userAgent=\&quot;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36\&quot; version=\&quot;9.1.8\&quot; editor=\&quot;www.draw.io\&quot; type=\&quot;google\&quot;&gt;&lt;diagram id=\&quot;2ae52b09-a662-b48f-7fa8-6262e6cde334\&quot; name=\&quot;Page-1\&quot;&gt;zVfbcpswEP0az7QPyXCzcR4TJ007004ydWeSPMpoDZoAokIYnK/vCgSYiy+TS9sXvDparaTdoyN5Yi+i4laQJPjBKYQTy6DFxL6eWJY7t/CrgK0GbA34gtEKMltgyV5Ag4ZGM0Yh7ThKzkPJki7o8TgGT3YwIgTPu25rHnZnTYgPA2DpkXCIPjAqgwqdT40W/wrMD+qZTUP3RKR21kAaEMrzHci+mdgLwbmsrKhYQKhyV+elGvdlT2+zMAGxPGWANZ3PiGub5mrueJTQMx1hQ8JMb3ZizUKMdbVCw1dGDaw5zoFbkFudl9nvjNcdZ2lZtUt0MC+Sou2sozwABjS+xRLEmnhQB8WlVnG7cyE8mH8lauTTMhEs9jFeFZUkScg8IhmPsZUzTLpl3HLu40It445kMiipgL9csJfS8fMwrC5Rsz8rD5iEZaKWa1/nSG10CmQUYstEk6RJxbY1K4A2ATYgJBR7C2Q2ZcfjAjwCKbboogc0nNdHxXR1O2+JZ841FuySrgaJJrvfxG75gIamxIn0sD+AHtMxevwEylIlEas3MgPStKJBKrnAg3lKnfFUJsr0tiGLKQj0OVL6Fc/QkX5fNQDxnn2h0LtMYhTQ+DtQwnG7lBhjhDXCiA8hxGyEEL1sepnYqONQ7h9ieqkUWGU3JFgbr5tIKJh8RNs4n+rWk2o1mQM6kOZe3nBungkPDgucJMIHeYjlw/zvJHg6kt8aExCiomy6yxxLup7hnrPypOw58Y7Vq1u1PT1qV9v7gWbdQLbbC1TlYBAI60O2O26Jckj3L9ju8dF0OncOGlXElmBNTl/HOfcDRGg+JkILfCmoVLxBgK5QB5Dzag0gNkxddEYumJSgNImpz/0WryFlyIDIkmbl9n2IQRAJZQeUzxQhs1hZEYoaSll6jvavoNRJPSgiz8ohS9WXr5UZs6J8ClVbMUi5mDZWJlnIpOpJBPcFicrH1OH5UbsUIf7H+9Lpk94aUcex09uA7yqP8+PyeFQRY1zEoxbBsvHU6ONf1ET3X2qi43Sral28UhOdi14g5zRNPC5b2Gyf7ZV7+9/HvvkD&lt;/diagram&gt;&lt;/mxfile&gt;&quot;}"></div> <script type="text/javascript" src="https://www.draw.io/js/viewer.min.js"></script> <h3 id="user-interface">User Interface</h3> <p>Spring boot comes with lot of convenient defaults that allows to jump start for developing any application. We will make use of spring boot security to configure our Oauth2 filter. We will make use of Google Oauth2 endpoint for authorizing our application. For this we first need to obtain the client key and client secret from Google. Go to <a href="https://start.spring.io/">Spring Initializr</a>. Select Web, security and oauth2 in the dependencies field and click on download button.</p> <h5 id="obtain-oauth2-client-key-and-client-secret-from-google">Obtain Oauth2 client key and client secret from Google</h5> <p>To obtain client key and client secret, Click on <a href="https://console.cloud.google.com">Google console</a>. Click on start new project from dropdown above, fill up the details of your demo project. Open the left side menu and select “API and Services” &gt; “Credentials”. Enter the required details in the form, Please note that for Spring “Authorised redirect URIs” should be “http://<servername>/login" e.g. http://localhost:8080/login. Click on create.</servername></p> <p>Once you generate the keys you are ready to use it in our Spring application.</p> <h5 id="configure-the-spring-application-with-oauth2-client-and-client-secret">Configure the Spring application with Oauth2 client and client secret</h5> <p>Before using the client and client secret we need to enable Oauth2 for Spring application. We will make use of <code class="language-plaintext highlighter-rouge">@EnableOAuth2Sso</code> annotation for this. This annotation allows us to configure OAuth endpoints using simple configuration in <code class="language-plaintext highlighter-rouge">application.yml</code> file.</p> <p>application.yml</p> <div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">security</span><span class="pi">:</span>
  <span class="na">oauth2</span><span class="pi">:</span>
    <span class="na">client</span><span class="pi">:</span>
      <span class="na">clientId</span><span class="pi">:</span> <span class="s">434972283788-j1n2hihmbr8hvf0a9hfa65dnvh15b4qh.apps.googleusercontent.com</span>
      <span class="na">clientSecret</span><span class="pi">:</span> <span class="s">gHPRImLetr6u-nT5lZTqeQm3</span>
      <span class="na">accessTokenUri</span><span class="pi">:</span> <span class="s">https://www.googleapis.com/oauth2/v4/token</span>
      <span class="na">userAuthorizationUri</span><span class="pi">:</span> <span class="s">https://accounts.google.com/o/oauth2/v2/auth</span>
      <span class="na">tokenName</span><span class="pi">:</span> <span class="s">access_token</span>
      <span class="na">authenticationScheme</span><span class="pi">:</span> <span class="s">query</span>
      <span class="na">clientAuthenticationScheme</span><span class="pi">:</span> <span class="s">form</span>
      <span class="na">scope</span><span class="pi">:</span> <span class="s">https://www.googleapis.com/auth/userinfo.profile</span>
    <span class="na">resource</span><span class="pi">:</span>
      <span class="na">userInfoUri</span><span class="pi">:</span> <span class="s">https://www.googleapis.com/oauth2/v2/userinfo</span>
<span class="na">app</span><span class="pi">:</span>
  <span class="na">service</span><span class="pi">:</span>
    <span class="na">cowsay</span><span class="pi">:</span>

<span class="na">logging</span><span class="pi">:</span>
  <span class="na">level</span><span class="pi">:</span>
    <span class="na">org.springframework.security</span><span class="pi">:</span> <span class="s">DEBUG</span>
</code></pre></div></div> <p>Spring boot application configuration</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@SpringBootApplication</span>
<span class="nd">@EnableOAuth2Sso</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">FortuneTellerUiApplication</span>  <span class="kd">extends</span> <span class="nc">WebSecurityConfigurerAdapter</span> <span class="o">{</span>

	<span class="nd">@Override</span>
	<span class="kd">protected</span> <span class="kt">void</span> <span class="nf">configure</span><span class="o">(</span><span class="nc">HttpSecurity</span> <span class="n">http</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
		<span class="n">http</span><span class="o">.</span><span class="na">antMatcher</span><span class="o">(</span><span class="s">"/**"</span><span class="o">).</span><span class="na">authorizeRequests</span><span class="o">().</span><span class="na">antMatchers</span><span class="o">(</span><span class="s">"/"</span><span class="o">,</span><span class="s">"/login**"</span><span class="o">,</span><span class="s">"/webjars/**"</span><span class="o">,</span><span class="s">"/error/**"</span><span class="o">).</span><span class="na">permitAll</span><span class="o">().</span><span class="na">anyRequest</span><span class="o">().</span><span class="na">authenticated</span><span class="o">().</span><span class="na">and</span><span class="o">().</span><span class="na">csrf</span><span class="o">().</span><span class="na">csrfTokenRepository</span><span class="o">(</span><span class="nc">CookieCsrfTokenRepository</span><span class="o">.</span><span class="na">withHttpOnlyFalse</span><span class="o">()).</span><span class="na">and</span><span class="o">();</span>
	<span class="o">}</span>

	<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
		<span class="nc">SpringApplication</span><span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="nc">FortuneTellerUiApplication</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">args</span><span class="o">);</span>
	<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div> <p>For calling the backend service we will make use of simple RestTemplate.</p> <h3 id="cowsay-rest-service">Cowsay rest service</h3> <p>We will develop our backend in python with <a href="https://www.djangoproject.com/">Django</a>. You need to have python 3.5 with pip installed on your machine to proceed further.</p> <h4 id="install-and-configure-django">Install and configure Django</h4> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Initialize project directory</span>
<span class="nb">mkdir </span>cowsay-service

<span class="c">## Create python virtual environment</span>
virtualenv <span class="nb">env
source env</span>/bin/activate

<span class="c"># Install Django and Django REST framework into the virtualenv</span>
pip <span class="nb">install </span>django
pip <span class="nb">install </span>djangorestframework

<span class="c"># Set up a new project with a single application</span>
django-admin.py startproject cowsay <span class="nb">.</span>  <span class="c"># Note the trailing '.' character</span>
<span class="nb">cd </span>cowsay
django-admin.py startapp cowsayRest
<span class="nb">cd</span> ..

</code></pre></div></div> <p>Create model for Message</p> <p>Update <code class="language-plaintext highlighter-rouge">cowsayRest/models.py</code> with our Message model</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Message</span><span class="p">(</span><span class="n">models</span><span class="p">.</span><span class="n">Model</span><span class="p">):</span>
    <span class="n">created</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="nc">DateTimeField</span><span class="p">(</span><span class="n">auto_now_add</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
    <span class="n">message</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="nc">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">2048</span><span class="p">)</span>
    <span class="n">serverName</span> <span class="o">=</span> <span class="n">models</span><span class="p">.</span><span class="nc">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">64</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">message</span><span class="p">,</span> <span class="n">serverName</span><span class="p">):</span>
        <span class="n">self</span><span class="p">.</span><span class="n">message</span> <span class="o">=</span> <span class="n">message</span>
        <span class="n">self</span><span class="p">.</span><span class="n">serverName</span> <span class="o">=</span> <span class="n">serverName</span>

    <span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
        <span class="n">ordering</span> <span class="o">=</span> <span class="p">(</span><span class="sh">'</span><span class="s">created</span><span class="sh">'</span><span class="p">,)</span>
</code></pre></div></div> <p>We need model to be serialzed to Json object so we need to define which fields to be included in the serialization and how the individual fields to be serialized.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MessageSerializer</span><span class="p">(</span><span class="n">serializers</span><span class="p">.</span><span class="n">Serializer</span><span class="p">):</span>
    <span class="n">message</span> <span class="o">=</span> <span class="n">serializers</span><span class="p">.</span><span class="nc">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">2048</span><span class="p">)</span>
    <span class="n">serverName</span> <span class="o">=</span> <span class="n">serializers</span><span class="p">.</span><span class="nc">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">64</span><span class="p">)</span>
</code></pre></div></div> <p>We will define view for our application which will just produce fortune and cowsay command to generate the message text, and this message text will be serialized using serializer defined above.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">fortune</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
    <span class="n">fortune</span> <span class="o">=</span> <span class="n">subprocess</span><span class="p">.</span><span class="nc">Popen</span><span class="p">((</span><span class="sh">'</span><span class="s">fortune</span><span class="sh">'</span><span class="p">),</span> <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="p">.</span><span class="n">PIPE</span><span class="p">)</span>
    <span class="n">output</span> <span class="o">=</span> <span class="n">subprocess</span><span class="p">.</span><span class="nf">check_output</span><span class="p">((</span><span class="sh">'</span><span class="s">cowsay</span><span class="sh">'</span><span class="p">),</span> <span class="n">stdin</span><span class="o">=</span><span class="n">fortune</span><span class="p">.</span><span class="n">stdout</span><span class="p">)</span>
    <span class="n">fortune</span><span class="p">.</span><span class="nf">wait</span><span class="p">()</span>
    <span class="n">message</span> <span class="o">=</span> <span class="nc">MessageSerializer</span><span class="p">(</span><span class="nc">Message</span><span class="p">(</span><span class="n">message</span> <span class="o">=</span> <span class="n">output</span><span class="p">.</span><span class="nf">decode</span><span class="p">(</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">),</span><span class="n">serverName</span> <span class="o">=</span> <span class="n">socket</span><span class="p">.</span><span class="nf">gethostname</span><span class="p">()))</span>
    <span class="k">return</span> <span class="nc">JsonResponse</span><span class="p">(</span><span class="n">message</span><span class="p">.</span><span class="n">data</span><span class="p">,</span> <span class="n">safe</span> <span class="o">=</span> <span class="bp">False</span><span class="p">)</span>
</code></pre></div></div> <p>We are all set only thing remaining is mapping the url to point to view we defined. initialize <code class="language-plaintext highlighter-rouge">cowsayRest/urls.py</code> with the url mapping as below</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span>
    <span class="nf">url</span><span class="p">(</span><span class="sa">r</span><span class="sh">'</span><span class="s">^fortune/$</span><span class="sh">'</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">fortune</span><span class="p">),</span>
    <span class="nf">url</span><span class="p">(</span><span class="sa">r</span><span class="sh">'</span><span class="s">^</span><span class="sh">'</span><span class="p">,</span> <span class="n">views</span><span class="p">.</span><span class="n">fortune</span><span class="p">),</span>
<span class="p">]</span>
</code></pre></div></div> <p>As you have noticed we have mapped <code class="language-plaintext highlighter-rouge">/fortune</code> and <code class="language-plaintext highlighter-rouge">/*</code> url to same view fortune. I have done this because there is a possibility that the load balancer check the health of the application by sending the request at the root of the application. Otherwise you can skip the second mapping and only keep the mapping for <code class="language-plaintext highlighter-rouge">fortune/</code></p> <p>Last but not the least we need to enable Django rest app in <code class="language-plaintext highlighter-rouge">cowsay/settings.py</code></p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">INSTALLED_APPS</span> <span class="o">=</span> <span class="p">[</span>
    <span class="sh">'</span><span class="s">rest_framework</span><span class="sh">'</span><span class="p">,</span>
 <span class="bp">...</span>
</code></pre></div></div> <h4 id="building-the-docker-containers">Building the docker containers</h4> <p>As we are making of Spring Boot, we will make use of plugin offered by Spring Framework to generate the docker image, so include below plugin in pom.xml</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;plugin&gt;</span>
				<span class="nt">&lt;groupId&gt;</span>com.spotify<span class="nt">&lt;/groupId&gt;</span>
				<span class="nt">&lt;artifactId&gt;</span>dockerfile-maven-plugin<span class="nt">&lt;/artifactId&gt;</span>
				<span class="nt">&lt;version&gt;</span>1.4.4<span class="nt">&lt;/version&gt;</span>
				<span class="nt">&lt;configuration&gt;</span>
					<span class="nt">&lt;repository&gt;</span>${docker.image.prefix}/${project.artifactId}<span class="nt">&lt;/repository&gt;</span>
					<span class="nt">&lt;buildArgs&gt;</span>
						<span class="nt">&lt;JAR_FILE&gt;</span>target/${project.build.finalName}.jar<span class="nt">&lt;/JAR_FILE&gt;</span>
						<span class="nt">&lt;tag&gt;</span>${project.artifactId}-${project.version}<span class="nt">&lt;/tag&gt;</span>
					<span class="nt">&lt;/buildArgs&gt;</span>
				<span class="nt">&lt;/configuration&gt;</span>
<span class="nt">&lt;/plugin&gt;</span>
</code></pre></div></div> <p>Create a Dockerfile under UI project</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
</code></pre></div></div> <p>For Cowsay rest service we will make use of Docker file as below</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM python:3
COPY . /usr/src/app
WORKDIR /usr/src/app
RUN pip install -r requirements.txt
EXPOSE 8000
USER root
RUN apt-get update &amp;&amp; apt-get -y --fix-missing upgrade &amp;&amp; apt-get install -y --fix-missing fortune cowsay
CMD ["./start.sh"]
</code></pre></div></div> <p>As you might have noted that we are using start.sh to initialize our Django app and start the development server</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export PATH=/usr/games:$PATH
python manage.py migrate
python manage.py runserver 0.0.0.0:8000 &gt;&gt; log 2&gt;&amp;1
</code></pre></div></div> <p>In the abo file we are initializing the PATH environment variable to include fortune and cowsay programs from <code class="language-plaintext highlighter-rouge">/usr/games/</code> directory.</p> <p>We will trigger the build for above two docker images using <code class="language-plaintext highlighter-rouge">build.sh</code></p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">cd </span>fortune-teller-ui<span class="p">;</span> ./mvnw clean package <span class="o">&amp;&amp;</span> ./mvnw <span class="nb">install </span>dockerfile:build<span class="o">)</span>
<span class="o">(</span><span class="nb">cd </span>cowsay-service/cowsay<span class="p">;</span> docker build <span class="nt">-t</span> spsarolkar/cowsay .<span class="o">)</span>
</code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">(cd fortune-teller-ui; ./mvnw clean package &amp;&amp; ./mvnw install dockerfile:build)</code> – This comand builds the User Interface executable jar file using spring boot maven plugin and then spring docker plugin generates the docker image.</p> <p><code class="language-plaintext highlighter-rouge">(cd cowsay-service/cowsay; docker build -t spsarolkar/cowsay .)</code> –&gt; This is a used for generating docker image for Cowsay Rest service</p> <h4 id="docker-swarm-configuration">Docker swarm configuration</h4> <p>Once we have both the images build we will move on the creating the docker Swarm configuration. create new file <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> under root of the project.</p> <div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3.2"</span>

<span class="na">services</span><span class="pi">:</span>
  <span class="na">sessions</span><span class="pi">:</span>
    <span class="na">image</span><span class="pi">:</span> <span class="s">redis:4</span>
    <span class="na">ports</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">6379:6379</span>
    <span class="na">networks</span><span class="pi">:</span>
      <span class="na">web</span><span class="pi">:</span>
        <span class="na">aliases</span><span class="pi">:</span>
          <span class="pi">-</span> <span class="s">redis</span>

  <span class="na">cowsay-service</span><span class="pi">:</span>
    <span class="na">image</span><span class="pi">:</span> <span class="s">spsarolkar/cowsay</span>
    <span class="na">ports</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="m">8000</span>
    <span class="na">environment</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">SERVICE_PORTS=8000</span>
    <span class="na">deploy</span><span class="pi">:</span>
      <span class="na">replicas</span><span class="pi">:</span> <span class="m">5</span>
      <span class="na">restart_policy</span><span class="pi">:</span>
        <span class="na">condition</span><span class="pi">:</span> <span class="s">on-failure</span>
        <span class="na">max_attempts</span><span class="pi">:</span> <span class="m">3</span>
        <span class="na">window</span><span class="pi">:</span> <span class="s">120s</span>
    <span class="na">networks</span><span class="pi">:</span>
      <span class="na">web</span><span class="pi">:</span>
        <span class="na">aliases</span><span class="pi">:</span>
          <span class="pi">-</span> <span class="s">cowsay-cluster</span>

  <span class="na">cowsay-proxy</span><span class="pi">:</span>
    <span class="na">image</span><span class="pi">:</span> <span class="s">dockercloud/haproxy:1.6.7</span>
    <span class="na">depends_on</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">cowsay-service</span>
    <span class="na">environment</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">BALANCE=leastconn</span>
    <span class="na">volumes</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">/var/run/docker.sock:/var/run/docker.sock</span>
    <span class="na">ports</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s2">"</span><span class="s">8000:80"</span>
    <span class="na">networks</span><span class="pi">:</span>
      <span class="na">web</span><span class="pi">:</span>
        <span class="na">aliases</span><span class="pi">:</span>
          <span class="pi">-</span> <span class="s">cowsay</span>
    <span class="na">deploy</span><span class="pi">:</span>
      <span class="na">placement</span><span class="pi">:</span>
        <span class="na">constraints</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">node.role == manager</span><span class="pi">]</span>

  <span class="na">cowsay-ui</span><span class="pi">:</span>
    <span class="na">image</span><span class="pi">:</span> <span class="s">spsarolkar/cowsay-ui</span>
    <span class="na">ports</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s2">"</span><span class="s">8080:8080"</span>
    <span class="na">environment</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">COWSAY_SERVER_NAME=cowsay</span>
      <span class="pi">-</span> <span class="s">COWSAY_SERVER_PORT=80</span>
      <span class="pi">-</span> <span class="s">REDIS_SERVER_NAME=redis</span>
      <span class="pi">-</span> <span class="s">REDIS_SERVER_PORT=6379</span>
    <span class="na">depends_on</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">redis</span>
      <span class="pi">-</span> <span class="s">cowsay</span>
    <span class="na">networks</span><span class="pi">:</span>
      <span class="na">web</span><span class="pi">:</span>

<span class="na">networks</span><span class="pi">:</span>
  <span class="na">web</span><span class="pi">:</span>
</code></pre></div></div> <p>Along with the images we built, there are two more images configured. Please find the details below.</p> <h5 id="redis4">redis:4</h5> <p>This image is used for redis database for session management. Currently we only configured single instance of UI but we can scale more just by adding more instances without worrying about sticky sessions.</p> <h5 id="dockercloudhaproxy167">dockercloud/haproxy:1.6.7</h5> <p>This image is a load balancer for backend Cowsay Rest service.</p> <p>We are all set to start our service using below command</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker stack deploy <span class="nt">--compose-file</span><span class="o">=</span>docker-compose.yml prod
</code></pre></div></div> <p>Once all images deployed you will see out UI once you visit the browser http://127.0.0.1:8080/</p> <p><img src="/assets/blog/BuildingMicroservicesUsingDocker/1_docker_swarm_example_landing_page.png" alt="landing-page"/></p> <p>Click on “Login with Google” button and you will be directed to Google OAuth authentication page. Once you login to your google account you will be redirected to home page and the <code class="language-plaintext highlighter-rouge">fortune</code> message will be displayed with <code class="language-plaintext highlighter-rouge">cowsay</code> decoration. Also notice the servername displayed in red to confirm that any subsequent requests are responded using different backend server handled by <code class="language-plaintext highlighter-rouge">haproxy</code> for load balancing purpose.</p> <p><img src="/assets/blog/BuildingMicroservicesUsingDocker/2_Rest_Server_Response.png" alt="cowsay-message-1"/></p> <p><img src="/assets/blog/BuildingMicroservicesUsingDocker/3_Rest_Server_Response.png" alt="cowsay-message-2"/></p>]]></content><author><name></name></author><category term="docker"/><category term="microservices"/><summary type="html"><![CDATA[Today we will learn how to build simple multi tier microservice using Oauth2 authorization with Redis session management. Our front end will be build using Spring Boot which will use backend python service for fetching the fortune message to be displayed. The block diagram for the application is as below. All the example code is uploaded to github at Github]]></summary></entry><entry><title type="html">Understanding Spring Boot Security configuration</title><link href="https://spsarolkar.github.io/blog/2018/Understanding-Spring-Boot-Security-Configuration/" rel="alternate" type="text/html" title="Understanding Spring Boot Security configuration"/><published>2018-09-09T09:18:36+00:00</published><updated>2018-09-09T09:18:36+00:00</updated><id>https://spsarolkar.github.io/blog/2018/Understanding-Spring-Boot-Security-Configuration</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2018/Understanding-Spring-Boot-Security-Configuration/"><![CDATA[<p>Spring boot came with many useful features and for most of the usual use cases we have all implementation already provided by Spring its just a matter of configuring it that suite our needs. One of the feature is spring security. I would assume that you have basic understanding of how Servlet Containers work. We will start with creating the sample application from <a href="https://start.spring.io">Spring Initializr</a>. Make sure you select Web and Security under “Search for dependencies” field. Once you download the zip file you will have standalone spring-boot application ready. You can simply run it using maven with command <code class="language-plaintext highlighter-rouge">mvn spring-boot:run</code></p> <p>You would notice that the application does not came with any <code class="language-plaintext highlighter-rouge">web.xml</code> file which was the main file that was used for configuring the DispatcherServlet used by Spring, this is because new Servlet API provides interface <code class="language-plaintext highlighter-rouge">ServletContainerInitializer</code>. Any class that implements this interface can configure the all the Servlet configuration on the fly, this allows Spring to hook up the new Filters and Servlet to be used by application on the fly. This feature allows configuration driven filter proxy to be used for authorization and authentication. We will see useful classes and intefaces defined by Spring that makes authentication and authorization.</p> <p>Any configuration for spring-security starts with extending <code class="language-plaintext highlighter-rouge">WebSecurityConfigurerAdapter</code>, this abstract class provides convenient methods for configuring spring security configuration using <code class="language-plaintext highlighter-rouge">HTTPSecurity</code> object. We will see how to use HTTPSecurity to configure authentication and authorization for our application below</p> <h5 id="restricting-the-httpsecurity-to-only-specific-urls">Restricting the HTTPSecurity to only specific urls</h5> <p>Consider the case that part of the web application urls are publicly accessible and part needs any special access. To achive this purpose HTTPSecurity provides methods as mentioned below</p> <h6 id="1-requestmatcher--to-enable-httpsecurity-for-specific-url-and-specific-http-method">1. requestMatcher : To enable <code class="language-plaintext highlighter-rouge">HTTPSecurity</code> for specific url and specific http method</h6> <p>If you want to restrict HTTPSSecurity only for specific url pattern then we have to use requestMatcher.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.requestMatcher(new AntPathRequestMatcher("/restricted/**",HttpMethod.POST.toString())).authorizeRequests().anyRequest().authenticated();
    }
</code></pre></div></div> <p>Above code will authorize any <code class="language-plaintext highlighter-rouge">POST</code> requested that is authenticated matching the pattern <code class="language-plaintext highlighter-rouge">/restricted/**</code></p> <h6 id="2-antmatcher--to-enable-httpsecurity-for-specific-url-pattern-irrespective-of-specific-http-method">2. antMatcher : To enable <code class="language-plaintext highlighter-rouge">HTTPSecurity</code> for specific url pattern irrespective of specific http method</h6> <p>If we dont need to restrict any specific HTTP method then we can make use of <code class="language-plaintext highlighter-rouge">antMatcher</code> as below</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http.antMatcher("/restricted/**").authorizeRequests().anyRequest().authenticated();
</code></pre></div></div> <h6 id="3-requestmatchers--to-enable-httpsecurity-for-multiple-url-pattern-for-multiple-http-method">3. requestMatchers : To enable <code class="language-plaintext highlighter-rouge">HTTPSecurity</code> for multiple url pattern for multiple http method</h6> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http.requestMatchers().
                antMatchers(HttpMethod.GET,"/restricgted/get/**","/restricgted2/get/**").
                antMatchers(HttpMethod.POST,"/restricgted/post/**","/restricgted2/post/**").
                and().authorizeRequests().anyRequest().authenticated();
</code></pre></div></div> <p>Above configuration will enable HTTPSecurity for any GET request with url patterns “/restricgted/get/<strong>” and “/restricgted2/get/</strong>” and any POST method with patterns “/restricgted/post/<strong>” and “/restricgted2/post/</strong>”</p> <p>All above cases will trigger <code class="language-plaintext highlighter-rouge">HTTPSecurity</code> for specific matchers, please note that <code class="language-plaintext highlighter-rouge">authorizeRequests()</code> method used above returns the <code class="language-plaintext highlighter-rouge">HTTPsecurity</code> object itself so its possible to skip <code class="language-plaintext highlighter-rouge">requestMatchers</code> or <code class="language-plaintext highlighter-rouge">antMatcher</code> configuration and streight away call <code class="language-plaintext highlighter-rouge">authorizeRequests</code> which will by default disable any filtering and enable HTTPSecurity for all the urls instead of filtered above.</p> <h5 id="lets-dig-deeper-into-the-important-interfaces-come-into-picture-for-spring-security-authentication">Lets dig deeper into the important interfaces come into picture for Spring Security Authentication</h5> <p>Spring Security relies on Servlet API Filters for controlling the flow for authentication. The main interface that is responsible for Authentication is <code class="language-plaintext highlighter-rouge">AuthenticationManager</code>. This interface provides single <code class="language-plaintext highlighter-rouge">authenticate</code> method. This method validates the credentials from <code class="language-plaintext highlighter-rouge">Authentication</code> object passed. Authentication manager generally chose whether it can return the Authentication object with <code class="language-plaintext highlighter-rouge">authenticated="true"</code> or it can throw <code class="language-plaintext highlighter-rouge">AuthenticationException</code> if the credentials do not match. Authentication manager cannot decide it should return <code class="language-plaintext highlighter-rouge">null</code>.</p> <p><code class="language-plaintext highlighter-rouge">AuthenticationManager</code> has multiple implementations of <code class="language-plaintext highlighter-rouge">ProviderManager</code>, the provider manager passes on the <code class="language-plaintext highlighter-rouge">Authentication</code> through the chain of <code class="language-plaintext highlighter-rouge">AuthenticationProvider</code>. Once any <code class="language-plaintext highlighter-rouge">AuthenticationProvider</code> returns the non null value the Authentication is marked as completed. Actual authentication happens in <code class="language-plaintext highlighter-rouge">AuthenticationProvider</code> implementation. To hook up custom <code class="language-plaintext highlighter-rouge">AuthenticationProvider</code> the <code class="language-plaintext highlighter-rouge">WebSecurityConfigurerAdapter</code> provides helper method</p> <p>The authentication filter has knowledge of which <code class="language-plaintext highlighter-rouge">AuthenticationManager</code> to use for authentication and during filter initialisation this AuthenticationManager also gets initialised. When we configure the httpSecurity object via <code class="language-plaintext highlighter-rouge">WebSecurityConfigurerAdapter</code> by default <code class="language-plaintext highlighter-rouge">AuthenticationFilter</code> gets initialised which makes use of <code class="language-plaintext highlighter-rouge">AuthenticationManager</code> with Username and Password authentication.</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Override</span>
<span class="kd">public</span> <span class="nf">configure</span><span class="o">(</span><span class="nc">AuthenticationManagerBuilder</span> <span class="n">builder</span><span class="o">)</span>
</code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">AuthenticationManagerBuilder</code> has <code class="language-plaintext highlighter-rouge">authenticationProvider</code> method that is used to set the local authentication provider for specific If you provide the override this will configure the <code class="language-plaintext highlighter-rouge">local</code> AuthenticationProvider that is specific to <code class="language-plaintext highlighter-rouge">WebSecurityConfigurerAdapter</code> implementation. For all usual use cases this would be sufficient. If you want to configure it for global, the we need inject global <code class="language-plaintext highlighter-rouge">AuthenticationManagerBuilder</code> and call its <code class="language-plaintext highlighter-rouge">authenticationProvider</code> method.</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">@Override</span>
<span class="kd">public</span> <span class="nf">configure</span><span class="o">(</span><span class="nc">AuthenticationManagerBuilder</span> <span class="n">builder</span><span class="o">)</span>
</code></pre></div></div> <p>We can provide multiple implementations of <code class="language-plaintext highlighter-rouge">WebSecurityConfigurerAdapter</code> in the single application to configure separate filter matching the specific <code class="language-plaintext highlighter-rouge">requestMatchers</code> and <code class="language-plaintext highlighter-rouge">antMatcher</code>. In every implementation we can configure seperate <code class="language-plaintext highlighter-rouge">authenticationProvider</code> using overriden <code class="language-plaintext highlighter-rouge">public configure(AuthenticationManagerBuilder builder)</code>.</p>]]></content><author><name></name></author><category term="spring-boot"/><category term="spring-security"/><summary type="html"><![CDATA[Spring boot came with many useful features and for most of the usual use cases we have all implementation already provided by Spring its just a matter of configuring it that suite our needs. One of the feature is spring security. I would assume that you have basic understanding of how Servlet Containers work. We will start with creating the sample application from Spring Initializr. Make sure you select Web and Security under “Search for dependencies” field. Once you download the zip file you will have standalone spring-boot application ready. You can simply run it using maven with command mvn spring-boot:run]]></summary></entry><entry><title type="html">Maven cheatsheet</title><link href="https://spsarolkar.github.io/blog/2018/Maven-cheatsheet/" rel="alternate" type="text/html" title="Maven cheatsheet"/><published>2018-09-01T12:32:15+00:00</published><updated>2018-09-01T12:32:15+00:00</updated><id>https://spsarolkar.github.io/blog/2018/Maven-cheatsheet</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2018/Maven-cheatsheet/"><![CDATA[<p>Maven has many different plugins and many times its very time consuming to google and find the suitable configuration to achieve your goal. Fortunately maven itself comes with help plugin which allows you to fetch all the necessary information about any plugin. As a example I will use the Spring boot plugin. The project you can easily obtain from <a href="https://start.spring.io/">Spring Initializr</a>.</p> <h4 id="get-the-list-of-goals-available-for-the-plugin">Get the list of goals available for the plugin</h4> <p>Our obvious starting point would be check what goals available for the plugin, below is the command to achieve the same</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn <span class="nb">help</span>:describe <span class="nt">-DgroupId</span><span class="o">=</span>org.springframework.boot <span class="nt">-DartifactId</span><span class="o">=</span>spring-boot-maven-plugin
</code></pre></div></div> <p>Output:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>INFO] Scanning <span class="k">for </span>projects...
<span class="o">[</span>INFO]
<span class="o">[</span>INFO] <span class="nt">------------------</span>&lt; org.apache.maven:standalone-pom <span class="o">&gt;</span><span class="nt">-------------------</span>
<span class="o">[</span>INFO] Building Maven Stub Project <span class="o">(</span>No POM<span class="o">)</span> 1
<span class="o">[</span>INFO] <span class="nt">--------------------------------</span><span class="o">[</span> pom <span class="o">]</span><span class="nt">---------------------------------</span>
<span class="o">[</span>INFO]
<span class="o">[</span>INFO] <span class="nt">---</span> maven-help-plugin:3.1.0:describe <span class="o">(</span>default-cli<span class="o">)</span> @ standalone-pom <span class="nt">---</span>
<span class="o">[</span>INFO] org.springframework.boot:spring-boot-maven-plugin:2.0.4.RELEASE

Name: Spring Boot Maven Plugin
Description: Spring Boot Maven Plugin
Group Id: org.springframework.boot
Artifact Id: spring-boot-maven-plugin
Version: 2.0.4.RELEASE
Goal Prefix: spring-boot

This plugin has 6 goals:

spring-boot:build-info
  Description: Generate a build-info.properties file based the content of the
    current MavenProject.

spring-boot:help
  Description: Display <span class="nb">help </span>information on spring-boot-maven-plugin.
    Call mvn spring-boot:help <span class="nt">-Ddetail</span><span class="o">=</span><span class="nb">true</span> <span class="nt">-Dgoal</span><span class="o">=</span>&lt;goal-name&gt; to display
    parameter details.

spring-boot:repackage
  Description: Repackages existing JAR and WAR archives so that they can be
    executed from the <span class="nb">command </span>line using java <span class="nt">-jar</span><span class="nb">.</span> With <span class="nv">layout</span><span class="o">=</span>NONE can also
    be used simply to package a JAR with nested dependencies <span class="o">(</span>and no main
    class, so not executable<span class="o">)</span><span class="nb">.</span>

spring-boot:run
  Description: Run an executable archive application.

spring-boot:start
  Description: Start a spring application. Contrary to the run goal, this
    does not block and allows other goal to operate on the application. This
    goal is typically used <span class="k">in </span>integration <span class="nb">test </span>scenario where the application
    is started before a <span class="nb">test </span>suite and stopped after.

spring-boot:stop
  Description: Stop a spring application that has been started by the <span class="s1">'start'</span>
    goal. Typically invoked once a <span class="nb">test </span>suite has completed.

For more information, run <span class="s1">'mvn help:describe [...] -Ddetail'</span>

<span class="o">[</span>INFO] <span class="nt">------------------------------------------------------------------------</span>
<span class="o">[</span>INFO] BUILD SUCCESS
<span class="o">[</span>INFO] <span class="nt">------------------------------------------------------------------------</span>
<span class="o">[</span>INFO] Total <span class="nb">time</span>: 1.611 s
<span class="o">[</span>INFO] Finished at: 2018-09-01T18:17:41+05:30
<span class="o">[</span>INFO] <span class="nt">------------------------------------------------------------------------</span>
</code></pre></div></div> <p>You can see that it has given you all the goals available for the plugin. For example <code class="language-plaintext highlighter-rouge">spring-boot:run</code> is the plugin goal to be used when we want to run the Spring Application.</p> <h4 id="get-the-details-of-any-specific-goal">Get the details of any specific goal</h4> <p>If we need to check bindings of any specific goal to the maven lifecycle phases it can be done using <code class="language-plaintext highlighter-rouge">-Ddetail</code> and <code class="language-plaintext highlighter-rouge">-Dgoal=&lt;goal name&gt;</code>.</p> <p>Command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn <span class="nb">help</span>:describe <span class="nt">-Ddetail</span> <span class="nt">-Dgoal</span><span class="o">=</span>start <span class="nt">-DgroupId</span><span class="o">=</span>org.springframework.boot <span class="nt">-DartifactId</span><span class="o">=</span>spring-boot-maven-plugin
</code></pre></div></div> <p>Output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[INFO] Scanning for projects...
[INFO]
[INFO] ------------------&lt; org.apache.maven:standalone-pom &gt;-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-help-plugin:3.1.0:describe (default-cli) @ standalone-pom ---
[INFO] Mojo: 'spring-boot:start'
spring-boot:start
  Description: Start a spring application. Contrary to the run goal, this
    does not block and allows other goal to operate on the application. This
    goal is typically used in integration test scenario where the application
    is started before a test suite and stopped after.
  Implementation: org.springframework.boot.maven.StartMojo
  Language: java
  Bound to phase: pre-integration-test

  Available parameters:

    addResources (Default: false)
      User property: spring-boot.run.addResources
      Add maven resources to the classpath directly, this allows live in-place
      editing of resources. Duplicate resources are removed from target/classes
      to prevent them to appear twice if ClassLoader.getResources() is called.
      Please consider adding spring-boot-devtools to your project instead as it
      provides this feature and many more.

    agent
      User property: spring-boot.run.agent
      Path to agent jar. NOTE: the use of agents means that processes will be
      started by forking a new JVM.

    arguments
      User property: spring-boot.run.arguments
      Arguments that should be passed to the application. On command line use
      commas to separate multiple arguments.

    classesDirectory (Default: ${project.build.outputDirectory})
      Required: true
      Directory containing the classes and resource files that should be
      packaged into the archive.

    excludeArtifactIds
      User property: spring-boot.excludeArtifactIds
      Comma separated list of artifact names to exclude (exact match).
      Deprecated. as of 2.0.2 in favour of {@code excludes}

    excludeGroupIds
      User property: spring-boot.excludeGroupIds
      Comma separated list of groupId names to exclude (exact match).

    excludes
      User property: spring-boot.excludes
      Collection of artifact definitions to exclude. The Exclude element
      defines a groupId and artifactId mandatory properties and an optional
      classifier property.

    folders
      User property: spring-boot.run.folders
      Additional folders besides the classes directory that should be added to
      the classpath.

    fork
      User property: spring-boot.run.fork
      Flag to indicate if the run processes should be forked. fork is
      automatically enabled if an agent, jvmArguments or working directory are
      specified, or if devtools is present.

    includes
      User property: spring-boot.includes
      Collection of artifact definitions to include. The Include element
      defines a groupId and artifactId mandatory properties and an optional
      classifier property.

    jmxName
      The JMX name of the automatically deployed MBean managing the lifecycle
      of the spring application.

    jmxPort
      The port to use to expose the platform MBeanServer if the application
      needs to be forked.

    jvmArguments
      User property: spring-boot.run.jvmArguments
      JVM arguments that should be associated with the forked process used to
      run the application. On command line, make sure to wrap multiple values
      between quotes. NOTE: the use of JVM arguments means that processes will
      be started by forking a new JVM.

    mainClass
      User property: spring-boot.run.main-class
      The name of the main class. If not specified the first compiled class
      found that contains a 'main' method will be used.

    maxAttempts
      The maximum number of attempts to check if the spring application is
      ready. Combined with the 'wait' argument, this gives a global timeout
      value (30 sec by default)

    noverify
      User property: spring-boot.run.noverify
      Flag to say that the agent requires -noverify.

    profiles
      User property: spring-boot.run.profiles
      The spring profiles to activate. Convenience shortcut of specifying the
      'spring.profiles.active' argument. On command line use commas to separate
      multiple profiles.

    skip (Default: false)
      User property: spring-boot.run.skip
      Skip the execution.

    useTestClasspath (Default: false)
      User property: spring-boot.run.useTestClasspath
      Flag to include the test classpath when running.

    wait
      The number of milli-seconds to wait between each attempt to check if the
      spring application is ready.

    workingDirectory
      User property: spring-boot.run.workingDirectory
      Current working directory to use for the application. If not specified,
      basedir will be used. NOTE: the use of working directory means that
      processes will be started by forking a new JVM.


[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.082 s
[INFO] Finished at: 2018-09-01T18:26:51+05:30
[INFO] ------------------------------------------------------------------------
</code></pre></div></div> <p>As you noticed it also printed which phase this goal in our case its the start goal is bound to.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Bound to phase: pre-integration-test
</code></pre></div></div> <p>This line tells us that start goal will automatically get executed during <code class="language-plaintext highlighter-rouge">pre-integration-test</code> maven phase</p> <p>Similarly we have <code class="language-plaintext highlighter-rouge">stop</code> goal which is bound to <code class="language-plaintext highlighter-rouge">post-integration-test</code> maven lifecycle phase. So that during integration test the application will be turned on so that integration testing can be performed on it.</p> <h4 id="check-which-goals-executed-when-execution-of-any-command">Check which goals executed when execution of any command</h4> <p>If we need to check what goals are invoked by any specific command execution, we can use <code class="language-plaintext highlighter-rouge">-Dcmd</code> and pass it the exact command, For example below command will check what goals are executed when we execute <code class="language-plaintext highlighter-rouge">mvn spring-boot:run</code></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn help:describe -Dcmd=spring-boot:run -Ddetail
</code></pre></div></div> <p>Output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[INFO] Scanning for projects...
[INFO]
[INFO] ---------------&lt; io.github.spsarolkar:fortune-teller-ui &gt;---------------
[INFO] Building fortune-teller-ui 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-help-plugin:2.2:describe (default-cli) @ fortune-teller-ui ---
[INFO] 'spring-boot:run' is a plugin goal (aka mojo).
Mojo: 'spring-boot:run'
spring-boot:run
  Description: Run an executable archive application.
  Implementation: org.springframework.boot.maven.RunMojo
  Language: java
  Bound to phase: validate
  Before this mojo executes, it will call:
    Phase: 'test-compile'

  Available parameters:

    addResources (Default: false)
      User property: spring-boot.run.addResources
      Add maven resources to the classpath directly, this allows live in-place
      editing of resources. Duplicate resources are removed from target/classes
      to prevent them to appear twice if ClassLoader.getResources() is called.
      Please consider adding spring-boot-devtools to your project instead as it
      provides this feature and many more.

    agent
      User property: spring-boot.run.agent
      Path to agent jar. NOTE: the use of agents means that processes will be
      started by forking a new JVM.

    arguments
      User property: spring-boot.run.arguments
      Arguments that should be passed to the application. On command line use
      commas to separate multiple arguments.

    classesDirectory (Default: ${project.build.outputDirectory})
      Required: true
      Directory containing the classes and resource files that should be
      packaged into the archive.

    excludeArtifactIds
      User property: spring-boot.excludeArtifactIds
      Comma separated list of artifact names to exclude (exact match).
      Deprecated. as of 2.0.2 in favour of {@code excludes}

    excludeGroupIds
      User property: spring-boot.excludeGroupIds
      Comma separated list of groupId names to exclude (exact match).

    excludes
      User property: spring-boot.excludes
      Collection of artifact definitions to exclude. The Exclude element
      defines a groupId and artifactId mandatory properties and an optional
      classifier property.

    folders
      User property: spring-boot.run.folders
      Additional folders besides the classes directory that should be added to
      the classpath.

    fork
      User property: spring-boot.run.fork
      Flag to indicate if the run processes should be forked. fork is
      automatically enabled if an agent, jvmArguments or working directory are
      specified, or if devtools is present.

    includes
      User property: spring-boot.includes
      Collection of artifact definitions to include. The Include element
      defines a groupId and artifactId mandatory properties and an optional
      classifier property.

    jvmArguments
      User property: spring-boot.run.jvmArguments
      JVM arguments that should be associated with the forked process used to
      run the application. On command line, make sure to wrap multiple values
      between quotes. NOTE: the use of JVM arguments means that processes will
      be started by forking a new JVM.

    mainClass
      User property: spring-boot.run.main-class
      The name of the main class. If not specified the first compiled class
      found that contains a 'main' method will be used.

    noverify
      User property: spring-boot.run.noverify
      Flag to say that the agent requires -noverify.

    profiles
      User property: spring-boot.run.profiles
      The spring profiles to activate. Convenience shortcut of specifying the
      'spring.profiles.active' argument. On command line use commas to separate
      multiple profiles.

    skip (Default: false)
      User property: spring-boot.run.skip
      Skip the execution.

    useTestClasspath (Default: false)
      User property: spring-boot.run.useTestClasspath
      Flag to include the test classpath when running.

    workingDirectory
      User property: spring-boot.run.workingDirectory
      Current working directory to use for the application. If not specified,
      basedir will be used. NOTE: the use of working directory means that
      processes will be started by forking a new JVM.


[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.775 s
[INFO] Finished at: 2018-09-01T19:35:11+05:30
[INFO] ------------------------------------------------------------------------
</code></pre></div></div> <p>As we can see we get below message indicating test-compile will get executed when <code class="language-plaintext highlighter-rouge">spring-boot:run</code> is executed</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  Before this mojo executes, it will call:
    Phase: 'test-compile'
</code></pre></div></div>]]></content><author><name></name></author><category term="maven"/><category term="cheatsheet"/><summary type="html"><![CDATA[Maven has many different plugins and many times its very time consuming to google and find the suitable configuration to achieve your goal. Fortunately maven itself comes with help plugin which allows you to fetch all the necessary information about any plugin. As a example I will use the Spring boot plugin. The project you can easily obtain from Spring Initializr.]]></summary></entry><entry><title type="html">Preview of themes for Rouge syntax highlighter</title><link href="https://spsarolkar.github.io/blog/2018/Rouge-themes-preview/" rel="alternate" type="text/html" title="Preview of themes for Rouge syntax highlighter"/><published>2018-08-05T17:25:00+00:00</published><updated>2018-08-05T17:25:00+00:00</updated><id>https://spsarolkar.github.io/blog/2018/Rouge-themes-preview</id><content type="html" xml:base="https://spsarolkar.github.io/blog/2018/Rouge-themes-preview/"><![CDATA[<p>Rouge is a popular syntax highlighter written in Ruby to provide the capability of syntax highlighting for code written on html pages. Rouge supports many different themes. While implementing the same feature for my blog I realized that there is no preview page where I can see how these themes look like. So I took the challenge and written my own, its written using javascript and all the static code gets generated using Jekyll. Please follow this link - <a href="https://spsarolkar.github.io/rouge-theme-preview/">Rouge Syntax highlighter preview page</a></p> <p><img src="/assets/blog/RougeSyntaxHeighlighterPreviewPage/rouge_syntax_highlighter_preview_page_select_theme.png" alt="rouge_syntax_highlighter_preview_page_select_theme"/></p> <p>As you can see it shows all the themes available for Rouge, select any theme and after selection you will see the syntax highlighter applied for specific theme on the sample code.</p> <h4 id="code-implementation">Code implementation</h4> <p>Core of the code lies in the shell script <code class="language-plaintext highlighter-rouge">generateThemeYaml.sh</code> , this shell script has two jobs first is collect all themes supported by Rouge and store it in the <code class="language-plaintext highlighter-rouge">themes.yaml</code> file under <code class="language-plaintext highlighter-rouge">_data</code> directory and second being generate css file for each of the theme and store it under <code class="language-plaintext highlighter-rouge">css</code> directory with file name format <code class="language-plaintext highlighter-rouge">syntax-&lt;theme-name&gt;.css</code>.</p> <p>The <code class="language-plaintext highlighter-rouge">themes.yaml</code> look as below</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="s">base16</span>
<span class="pi">-</span> <span class="s">base16.dark</span>
<span class="pi">-</span> <span class="s">base16.light</span>
<span class="pi">-</span> <span class="s">base16.monokai</span>
</code></pre></div></div> <p>After generating <code class="language-plaintext highlighter-rouge">themes.yaml</code> and all stylesheets. We just need to iterate it in our Jekyll page using liquid syntax.</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {% for themename in site.data.themes %}
<span class="nt">&lt;a</span> <span class="na">class=</span><span class="s">"dropdown-item"</span> <span class="na">href=</span><span class="s">"javascript:reaplyStyles('{{themename}}')"</span><span class="nt">&gt;</span>{{ themename }}<span class="nt">&lt;/a&gt;</span>
{% endfor %} 
</code></pre></div></div> <p>Above code generated the dropdown list populated with all the themes. Once dropdown is selected its the simple Jquery to remove the existing themes stylesheet(if any) and add new stylesheet for the new theme selected.</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nf">removeAllSyntaxStyles</span><span class="p">()</span> <span class="p">{</span>
  <span class="nf">$</span><span class="p">(</span><span class="dl">'</span><span class="s1">link[rel=stylesheet][href*="syntax"]</span><span class="dl">'</span><span class="p">).</span><span class="nf">remove</span><span class="p">();</span>
<span class="p">}</span>

<span class="kd">function</span> <span class="nf">addStyle</span><span class="p">(</span><span class="nx">themename</span><span class="p">)</span> <span class="p">{</span>
  <span class="nf">$</span><span class="p">(</span><span class="dl">"</span><span class="s2">&lt;link&gt;</span><span class="dl">"</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">attr</span><span class="p">(</span><span class="dl">"</span><span class="s2">rel</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">stylesheet</span><span class="dl">"</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">attr</span><span class="p">(</span><span class="dl">"</span><span class="s2">type</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">text/css</span><span class="dl">"</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">attr</span><span class="p">(</span><span class="dl">"</span><span class="s2">href</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">css/syntax-</span><span class="dl">"</span> <span class="o">+</span> <span class="nx">themename</span> <span class="o">+</span> <span class="dl">"</span><span class="s2">.css</span><span class="dl">"</span><span class="p">)</span>
    <span class="p">.</span><span class="nf">appendTo</span><span class="p">(</span><span class="dl">"</span><span class="s2">head</span><span class="dl">"</span><span class="p">);</span>
<span class="p">}</span>

<span class="kd">function</span> <span class="nf">reaplyStyles</span><span class="p">(</span><span class="nx">themename</span><span class="p">)</span> <span class="p">{</span>
  <span class="nf">removeAllSyntaxStyles</span><span class="p">();</span>
  <span class="nf">addStyle</span><span class="p">(</span><span class="nx">themename</span><span class="p">);</span>
  <span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div> <p>I feel the utility is still at basic level and it does not allow selecting the custom background for the users who want to see how code looks for different background colours. This is a TODO for now.</p> <p>Please feel free to <a href="https://github.com/spsarolkar/rouge-theme-preview">fork the repository</a> if you feel adding any new feature. Please note that all the code is present under <code class="language-plaintext highlighter-rouge">gh-pages</code> branch</p>]]></content><author><name></name></author><category term="rouge"/><category term="themes"/><category term="syntax-highlighter"/><summary type="html"><![CDATA[Rouge is a popular syntax highlighter written in Ruby to provide the capability of syntax highlighting for code written on html pages. Rouge supports many different themes. While implementing the same feature for my blog I realized that there is no preview page where I can see how these themes look like. So I took the challenge and written my own, its written using javascript and all the static code gets generated using Jekyll. Please follow this link - Rouge Syntax highlighter preview page]]></summary></entry></feed>