Will Copilot pass a job interview with a grumpy old senior developer

So, here’s the setup: I took a typical example used in technical job interviews, and the candidate was Copilot. I set the Copilot model selection to Auto.

The problem presented in the interview was based on the following interface:

using System;
using System.Collections.Generic;
using System.Text;

namespace ClassLibrary
{
    internal interface IBatch<T>
    {
        /// <summary>
        /// Adds an item to internal storage
        /// </summary>
        /// <param name="item"></param>
        void AddItem(T item);

        /// <summary>
        /// Return a Batch of 100 items if storage holds at least 100 items.
        /// If oldest item in storage is older than 5 minutes, 
        /// return a batch of all items in storage.
        /// If number of items in storage is less than 100 and oldest item 
        /// is younger than 5 minutes, return empty collection.
        /// </summary>
        /// <returns></returns>
        IEnumerable<T> GetBatch();
    }
}

So my first prompt to copilot was:
Create an implementation of IBatch and follow the description in the interface definition. Be aware that both AddItem and GetBatch should be thread safe and will be called from a multithreaded application.

Here is the implementation Copilot came up with - and it was a lot faster than any humane could have typed it!

using System;
using System.Collections.Generic;

namespace ClassLibrary
{
    internal sealed class Batch<T> : IBatch<T>
    {
        private const int MaxBatchSize = 100;
        private static readonly TimeSpan MaxItemAge = TimeSpan.FromMinutes(5);

        private readonly Queue<BatchItem> _items = new();
        private readonly object _syncRoot = new();

        public void AddItem(T item)
        {
            lock (_syncRoot)
            {
                _items.Enqueue(new BatchItem(item, DateTime.UtcNow));
            }
        }

        public IEnumerable<T> GetBatch()
        {
            lock (_syncRoot)
            {
                if (_items.Count == 0)
                {
                    return Array.Empty<T>();
                }

                if (_items.Count >= MaxBatchSize)
                {
                    return DequeueBatch(MaxBatchSize);
                }

                var oldestItemAge = DateTime.UtcNow - _items.Peek().EnqueuedAtUtc;
                if (oldestItemAge > MaxItemAge)
                {
                    return DequeueBatch(_items.Count);
                }

                return Array.Empty<T>();
            }
        }

        private List<T> DequeueBatch(int count)
        {
            var batch = new List<T>(count);
            for (var i = 0; i < count; i++)
            {
                batch.Add(_items.Dequeue().Item);
            }

            return batch;
        }

        private sealed record BatchItem(T Item, DateTime EnqueuedAtUtc);
    }
}

Ok we have some code to dicuss - a starting point for all the interview question which will show if the candidate knows c#

---

Improve the code

Given the implementation Copilot came up with the first things I noticed was

• Batch is not testable due to the direct binding to DateTime.UtcNow
• Use of object instead of the new’er Lock object in .NET
• Using Queue and manually controlling the thread access instead off using ConcurrentQueue (it has been there for years)
• And then offcourse all the questions with memory and GC

So I gave Copilot the following propmt:
Review the Batch class and list potential issues

And here is what Copilot came up with:

Not what I was looking for

Surprised unittest wasn’t a concern and also the usage of object instead of the new Lock introduced in .Net9 and C#13 (I just looked it up, it was introduced November 2024 - almost 2 years ago)

I wanted to make the implementation more testable. I had to re-phrase the question a couple of times since Copilot confused itself because there was no unittest. This also happens during a normal job interview - the interviewer also have to be clear and produce good questions. So the following prompt:
The Batch class is not really testable. It will be difficult to write good unittests, why?

copilot:

So at the end Copilot came up with the right solution. But I don’t see any value in suggesting sleeps/waits - thats for sure a huge bummer.

What about performance

My initial question to Copilot about potential issues did not raise the concerns I had for memory, performance and GC. So instead of asking Copilot directly about performance and memory I asked more indirectly - as I would have done if I was talking to a human candidate. So i promted Copilot: Which input parameters would be suitable for a Batch constructor to best control the internals of the class

Copilots answer:

Then I prompted Copilot: What is the consequence of not specifying an initial capacity for the Queue

copilot:

ME: If the Batch class shall be used under heavy load and performance is a concern, which modifications should the be considered

copilot:

How will it impact performance and GC if I specify the type of T - e.g. class, record or struct

copilot:

Conclusion

I think the most important thing we can conclude is that you really need to know what to ask or prompt for.Copilot did come up with acceptable answers. But general questions won’t cut it—you have to be more specific. And to be specific, you have to know the fundamentals yourself.

This little exercise has made me a bit more skeptical about using Copilot for code reviews. Or at least, it’s something for my brain to chew on while I’m walking the dog.

I’ll keep my GitHub Copilot subscription. But I’m not going to use it unattended—no autopilot yet.