Written by Julie Zelenski
Welcome to the assignment advice/FAQ page!
The assignment writeup is the place to go for the formalities (required specifications, due date, grading standards, logistics, and so on). This is the companion advice page where we offer hints and recommendations and answer common question that we expect may come up along the way. You may want to skim this page now to get a sense of what's covered and return here on an as-needed basis for further review when/if these issues arise.
Advice
- A great place to begin is by reading our advice on effective software testing, specifically the section on black-box testing. Draft a testing plan of what behaviors of your program to examine and sketch out small inputs that cover the various cases.
- The bugs are designed so that you can reproduce them using an input of just a few fragments. The same trigger may also exist in some/all of the larger sample files, but it can be difficult to identify the pattern in such a sea of data. You'll narrow in on your issue more directly by building up small examples than by breaking down large ones.
- It can be convenient to enter input via the terminal, also known as
stdin(standard in). This allows you to run a quick test without the overhead of creating and saving the input to a file. Just invoke reassemble with no arguments, type the fragments directly at the terminal, and indicate you're finished by typing control-d on a line by itself.
Frequently asked questions about assign0
When I try to run the reassemble program in my directory, it responds "command not found". What's wrong?
Unix wants you to instead refer to the program by its full name ./reassemble. A longer answer and how to enable use of the short name can be found in our frequently asked questions about unix.
What is the difference between commit and submit? Do I need to do both?
Both are essential. hg commit takes a current snapshot of your work and saves it to your history. You will make many commits during development, but this history is stored only in your local repo. As a final step, submit copies your complete work into the class repo and makes it available to the staff for grading. If you only commit, but never submit, we never see your work and will score the empty repo as a zero. Be sure you both commit AND submit!
How do I use the sample executable? How does it relate to sanity check?
Our provided sample executable can be used a reference implementation during testing. Run the sample, e.g. slink/reassemble_soln slink/allswell_frags, to observe the expected behavior. Now run your program on the same input ./reassemble slink/allswell_frags. If your program produces the same result as the sample, all is good. A manual comparison becomes tedious as you have more/larger tests, so we wrote the sanity check tool to help. It automates the process of running a command once on the sample and again on your program, compare the two results, and report match/mismatch. Read our guide on how to use sanity check.
I ran the default sanity check on my buggy reassemble and it passed with flying colors. What gives?
The default sanity check runs a small set of very basic cases and reports on only and exactly those tests. Success on default sanity check means your program is functional enough to pass those simple tests, but doesn't confirm or deny anything beyond that. You'll need to devise your own test suite with more comprehensive coverage to expose lurking bugs. Your own tests can be used with the custom option of sanity check.
My program fails on input A, but my friend's program works fine on that input. How can that be?
The bugs are randomly seeded. Each program has one bug from each category but it is not necessarily the same bug as others have. Your program is a custom, artisanal artifact created just for you!
What is the expected behavior on fatal errors, such as incorrect usage or reading malformed input?
For fatal conditions, a program is expected to do three things: (1) detect the issue, (2) give the user a clear message about the problem, and (3) gracefully exit. A buggy program could fail to detect the problem at all, another might detect it but misreport it, and another might provide a great message but forget to exit and stumble on to a crash. Botching any of the three is a bug.
My program reports errors differently than the sample. Is this a bug?
Possibly. Any mismatch in the behavior/output of your program compared to the sample is a starting point for further investigation. It is not a problem if the program is merely using a different wording that is equivalently appropriate and informative. However, if its handling deviates from the required specification or the program's behavior is clearly worse than the sample (e.g. crashing instead of a clean exit or using a cryptic/inappropriate error message), it's a bug.
I've identified a bug, but not sure which area if affects. Can you further identify the divisions?
A problem validating the command-line arguments or getting the program off and running is a usage bug. If properly invoked and execution has begun, you're past usage. Next is reading the input and validating that it fits the required format. Any trouble during reading (e.g. wrongly rejecting valid input, mishandling malformed input) is a read bug. You can test read in isolation by invoking reassemble -r to stop after read and before reassembly. If the input is well-formed and read correctly, you're on to logic. If reassembly produces incorrect output or exhibits a runtime misbehavior (crash, hang, premature exit), it's a logic bug. If the program is correctly invoked on a valid file that reassembles to the correct result (no bug in usage, read, or logic) yet the valgrind reports shows an issue, that's a memory bug.
I think I've found two distinct bugs in the same area. What gives?
We planted only one bug per area, so if you are seeing more than one bad behavior in a single area, look for a larger pattern that encompasses both.
The spec says fragments can contain any character other than curly braces. Does that mean extended ASCII chars, such as é or π, too?
Yes, these are allowed by spec and should be reassembled correctly. However, these characters require jumping through hoops to enter in editors and can have odd interactions with your terminal on stdin, all of which makes them difficult to test. None of our planted bugs is specific to characters in the extended ASCII range. You will be able to reproduce all of your bugs without using those characters, so no need to look there for trouble.
My program seems to stop after reading 5,0000 fragments and silently discards the rest of the input. Is this a bug?
No. The specification does not require error detection/handling if the input contains more than the maximum number of fragments, so programs are free to do anything reasonable. This behavior is not the read bug you are looking for.
The trigger for my logic bug seems to be quite obscure. Could my understanding be off?
Perhaps. The flaws we've created are representative of ordinary coding errors/oversights and not designed to be intentionally esoteric. For example, a logic error might affect every containing match, but we would not insert a bug that triggers on a weird isolated case like merging a string of duplicate letters with a even-length palindrome. In particular, no logic bug will depend on the specific characters within the fragment, so it would be a red herring to fixate on the presence/absence of punctuation in the fragments, for example. The underlying pattern will always be straightforward to describe and based on general properties.
My program takes a long time to reassemble a large input and even longer to run under Valgrind. Is this slowness a bug?
No. The buggy programs all have performance on par with the sample and the reassembly process slows down on larger inputs. Running under Valgrind is even slower. All of this is normal and expected.
Is the Valgrind bug going to be a memory error or a memory leak?
The buggy programs have been seeded with a memory error. Read cs107 guide to Valgrind for more info on the difference between the two.
When my program exits due to fatal error (such as when reading malformed input), Valgrind reports memory leaks. Is this my memory bug?
No. The memory problem you are looking for is a memory error (not a leak) reading valid input that successfully reassembles. We are not looking for memory leaks in runs that exit prematurely due to error.
Am I responsible for identifying the root flaw in the code is or suggesting how the developer might fix it?
No. As a black-box outsider, you aren't privy to the necessary information to make this kind of judgment. When brainstorming test inputs, it may be helpful to speculate about possible bugs and how they would surface (e.g. an off-by-one loop bound), but the bug report needs only to identify the observed pattern and provide the reproducible test input.