• Speed Up Your Django Tests

    I read the book “Speed Up Your Django Tests” this week, a few interesting items: 

    Background/disclaimer: I’m new to Django,  I use pytest to run many integration Django tests. so the points listed here are purely from my point of view.

    1. Override settings: with @override_settingsin case you want to override a setting for a test method, Django provides the override_settings() decorator (see PEP 318).
    2. Show slow tests with pytest --durations 10
    3. Tests marker, categorize/tag tests so that can run different subsets.  like JUnit categories for more details: https://docs.pytest.org/en/latest/example/markers.html
    4. Reduce pytest test collection by setting norecursedirs
    5. Run in parallel with pytest-xdist
    6. Django’s RequestFactory: This is similar to the test client, but instead of making requests, “provides a way to generate a request instance that can be used as the first argument to any view” Django Doc
    7. Django’s SimpleTestCase:  a subclass of unittest.TestCase, it “disallows database queries by default.”,  however, you till can turn it on.
    8. Avoid Fixture Files[11.1],  “For data you need in individual tests, you’re better off creating it in the test case or test method.” I have to see it’s very easy to set up test data with fixtures, but shortly it becomes unmanageable few valid points: 

      Fixture ˉles are separate from the tests that use them. This makes it hard to determine which tests use which objects. The ˉles tend to become “append-only,”…when a new test needs a new object, it tends to be added to an existing file…if there’s some data that most of your application depends on, using a fixture, causes unnecessary reloading. It will be loaded and then rolled back for each test case, even when the next test case needs the exact same data.

    Overall, I would say it’s a good Django testing book for newbies like me, the book also covers many other topics, such as “Profiling”, “Mocking” etc, and many topics and links for me to explore Django. overall, I would say it’s a good Django testing book for newbies like me.

    However, slow tests generally indicate design issues. all the techniques mentioned in the book definitely can help to speed up the testing(itself), if we take one steps further, should we start thinking about the design?

    Abstraction

    from: Architecture Patterns with Python

    if we cloud fundamentally resolve some design issues, I believe we’ll get much fewer integration tests.

  • How to get a software engineering job in Singapore

    If you’re thinking about it, you might want to start with these two websites:

    1. MyCareersFuture.sg

    This is a portal that aims to provide Singapore Citizens and Permanent Residents with a fast and smart job search service to match them with relevant jobs… The portal was developed by Workforce Singapore, in partnership with the Government Technology Agency.

    This is not just another government website:

    From 1 Aug 2014, under the Fair Consideration Framework (FCF) by the Ministry of Manpower (MOM), companies seeking to hire Employment Pass (EP) holders are required to post their job vacancies on MyCareersFuture.sg for at least 14 calendar days before an EP application is submitted to MOM. For more information on FCF, click here.

    So technically speaking, the HR department will list the job vacancies as early as possible.

    This is not all, most important feature for me is the salary info, even though only a range, it really helps me understand the market (on some level), for example, by reading few lists here, I know that:

    Some of the packages may have different components, but the range gives me much better visibility of the market.

    2. efinancialcareers.sg

    As you may know that many financial institutions have development teams in Singapore, so you better set up your profile at efinancialcareers.sg — even if you have no interest in this industry.

    Setting a profile here will attract most of the recruitment agencies, they will try to talk to you, share their opportunities. Be open-minded, talk with them, they will try their best to find a matching opportunity for you.

    However, other websites such as “Linkedin”, “Indeed” are also very helpful, but for me, “MyCareersFuture” and “efinancialcareers” are the most effective ones.

    Good luck!

  • Google Cloud Tasks: use queue.xml to control rate for slow queues

    We got a service that has an HTTP request rate limit: less than 1 message per 10 seconds. we don’t use this service frequently, but when we use it, we send two requests sequentially, as expected, we received few http 429 errors.

    I kind of agree that it’s my responsibility to control the rate, but I don’t want to let my code aware of these constraints, so we decided to let Google Tasks control the rate.

    First try

    I don’t know much about token bucket, by glancing the help doc, I think it will help by setting --max-dispatches-per-second=0.01  (1 message / 10 seconds) with:

    cloud tasks queues update my-task-queue --max-concurrent-dispatches=1  --max-dispatches-per-second=0.01
    

    however, we noticed that HTTP 429 persists after the change, task queue log shows tasks are dispatched almost at the same time.  until we checked the maxBurstSize

     Each queue has a token bucket that holds tokens, up to the maximum specified by maxBurstSize. Each time a task is dispatched, a token is removed from the bucket. Tasks will be dispatched until the queue’s bucket runs out of tokens. The bucket will be continuously refilled with new tokens based on maxDispatchesPerSecond. and this field is an output value of gcloud, gcloud tasks queues describe my-task-queue shows the maxBurstSize is 10.

    so the bucket should have 10 tokens initially, even though I set the rate, but in my case, the first call will get run immediately because 10 tokens are available right there.  read the document again, and I found:

     In most cases, using the Cloud Tasks API method and letting the system set max_burst_size produces a very efficient rate for managing request bursts. In some cases, however, particularly when the desired rate is relatively slow, either using the queue.yaml method to manually set bucket_size to a small value, or setting your max_concurrent_dispatches to a small value via the Cloud Tasks API can give you more control. https://cloud.google.com/tasks/docs/configuring-queues#rate

    Second try

    set bucket_size to 1 using queue.yaml, task queue log shows tasks are dispatched right at the rate I set. 

    That’s not all,  you’d better to read this one before using queues.xml:

    Pitfalls of mixing queue.yaml and gcloud command

    and these posts also help:

  • Working at a startup for 6 months

    It has been (almost) 6 months with my new role at a startup, I feel like that I’m running everyday and have no time to really think about the change. Is this kind of the “new stupid”? I hope it’s not, so I want to summarize and write it down the difference I learned.

    I managed to concentrate on my work and “run as fast as I can”

    Well, I’m not saying that I didn’t concentrate on my work in my previous jobs. What I mean is I literally spend all my time on my work, not on meetings, emails, reports etc,. I used to spend a lot of time updating my manager on what I was doing and I needed to produce nice reports to my managers and sometimes, my manager’s managers.

    The bottleneck is not the infrastructure team or change management team, it’s just me. I have access to all products, I can deploy or destroy the production environment anytime, as long as I’m aware of what I’m doing. That’s cool!

    Delivery the product, not beautiful code

    I used to work in a relatively bigger team with ~20 developers, splitted to 3 prods focused on different products. Most of the banks I worked for have “change-freeze” periods, mostly December will be one of them. One day, my manager declared that we’ll do “platform enhancements” with a few nice diagrams. However, at the moment we all know that our product got few bugs like users cannot buy our products just because they don’t have a valid date of birth in the CRM.

    I’m not very happy with it because the proviritoy is wrong to me, if I receive 10% of each transaction, I’ll fix all the known production issues ASAP. but I’m not receiving any commision, neither does my manager. So we had a nice holiday season on refactoring. I enjoyed it, but I don’t think stakeholders will be happy if they really understand what we’re doing.

    Small startups are completely different, with limited resources, we need to deliver a working product and get actual feedback from the markets. They might like the product because of a nice design, but definitely not because of my beautify code.

    Size Does Matter

    I don’t really understand this when I first read it few years back, but now I do:

    We’re small and we like it that way. It gives us the ability to turn on a dime, deliver projects quickly, and dedicate extraordinary attention to your assignment. Our size allows us to work on projects we want to do rather than projects we have to do. Plus we can all fit in one cab if we squeeze. [https://37signals.com/05.html]

    Do everything and get the job done

    In my previous job, I and one of my teammates acted as the “DevOps” team for one year, there’s a big gap from code and functionally product, and nobody wants to get hands dirty to fix the gap. we decided to do everything required to deliver the code to a product, without writing much code for the product. we handle many tasks from managing stakeholder’s expectations, involved in system design (we gave up soon because of manager-driven design), code review, quality control, deployment, production support.

    the mindset is different here, we don’t have dedicated team to help you on database, messaging queue, network etc.  so in 6 months, I picked python and Django, got my hands dirty with Google cloud, delivered few features, and of course, fixed bugs created by myself. 

  • Lessons Learned from Take-Home-Projects in Job Interviews

    Consider a take-home-project an opportunity to demonstrate your design and coding skills. writing aFactory to produce intances is much more interesting than answering ‘can you tell me one design pattern that you used?’

    Prepare a seed/template project to save time

    e.g. https://github.com/guoliang-dev/java8-maven-seed-project:

    • a default pom.xml with dependencies(e.g. junit) and plugins(e.g. jaccoco)
    • a default CI config e.g. bitbucket
    • a default readme

    Put your solution into a remote private repo and setup build pipeline

    such as github/bitbuket, make sure it’s a private one and build pipeline is enabled. both github and bitbuket are offering certain free minutes for private repo build.

    Prepare a readme.md

    typically a readme file should contains:

    • your name, email and your github url
    • assumptions
    • high level design
    • TODO items: show that you can continue ehancing the project if you got more time

    however, don’t expect that interviewers will read all details carefully.

    Add comments in your code

    this will help interviewers understand your code and also let them know that you’re aware of poetnial issues.

    generally I’ll try my best on naming, so that I dont need to add comments. howerver, comments will help for:

    • complex codes
    • messy codes, e.g: // todo: refactoring required, should reduce the size of this method
    • shortcut. e.g: // todo: use proper ThreadPool to handle MT

    Unit tests / intergration tests / end-to-end tests

    always write unit tests, even though it’s not mentioned at all, ideally follow TDD.

    • tests help you verify your solution
    • tests help you make further changes safely
    • tests help you design your project in clean way

    Think about the next stpes

    you may required to have code review sessions with interviewers:

    • you need to share your design and might get challenged about the technical design you made
    • you might required to fix certain bugs or add features

    Coding during interview

    if you’re requested to fix a bug or add an enhancement, make sure:

    • run all your tests before making any change
    • limit your change scope: only make minimal necessary changes to delievery the task, you might want to refactor a lot DO NOT change anything if it’s not linked to the task.

subscribe via RSS