The Decimal Number System – Division, Rounding and Conversion

in Aviation Mathematics

Division of Decimal Numbers

Division of decimal numbers is performed the same way as whole numbers, unless the divisor is a decimal.

When the divisor is a decimal, it must be changed to a whole number before dividing. To do this, move the decimal in the divisor to the right until there are no decimal places. At the same time, move the decimal point in the dividend to the right the same number of places. Then divide. The decimal in the quotient will be placed directly above the decimal in the dividend.

Example: Divide 0.144 by 0.12

Move the decimal in the divisor (0.12) two places to the right. Next move the decimal in the dividend (0.144) two places to the right. Then divide. The result is 1.2.

Example: The wing area of an airplane is 262.6 square feet and its span is 40.4 feet. Find the mean chord of its wing using the formula: Area ÷ span = mean chord.

Move the decimal in the divisor (40.4) one place to the right. Next move the decimal in the dividend (262.6) one place to the right. Then divide. The mean chord length is 6.5 feet.

Rounding Off Decimal Numbers

Occasionally, it is necessary to round off a decimal number to some value that is practical to use.

For example, a measurement is calculated to be 29.4948 inches. To use this measurement, we can use the process of “rounding off.” A decimal is “rounded off” by keeping the digits for a certain number of places and discarding the rest. The degree of accuracy desired determines the number of digits to be retained. When the digit immediately to the right of the last retained digit is 5 or greater, round up by 1. When the digit immediately to the right of the last retained digit is less than 5, leave the last retained digit unchanged.

Example: An actuator shaft is 2.1938 inches in diameter. Round to the nearest tenth.

The digit in the tenths column is a 1. The digit to the right of the 1 is a 9. Since 9 is greater than or equal to 5, “round up” the 1 to a 2. Therefore, 2.1938 rounded to the nearest tenth is 2.2.

Example: The outside diameter of a bearing is 2.1938 inches. Round to the nearest hundredth.

The digit in the hundredths column is a 9. The digit to the right of the 9 is a 3. Since 3 is less than 5, do not round up the 9. Therefore, 2.1938 to the nearest hundredth is 2.19.

Example: The length of a bushing is 2.1938 inches. Round to the nearest thousandth.

The digit in the thousandths column is a 3. The digit to the right of the 3 is an 8. Since 8 is greater than or equal to 5, “round up” the 3 to a 4. Therefore, 2.1938 to the nearest thousandth is 2.194.

Converting Decimal Numbers to Fractions

To change a decimal number to a fraction, “read” the decimal, and then write it into a fraction just as it is read as shown below.

Example: One oversized rivet has a diameter of 0.52 inches. Convert 0.52 to a fraction. The decimal 0.52 is read as “fifty-two hundredths.”


A dimension often appears in a maintenance manual or on a blueprint as a decimal instead of a fraction. In order to use the dimension, it may need to be converted to a fraction. An aviation mechanic frequently uses a steel rule that is calibrated in units of 1⁄64 of an inch.  To change a decimal to the nearest equivalent common fraction, multiply the decimal by 64. The product of the decimal and 64 will be the numerator of the fraction and 64 will be the denominator. Reduce the fraction, if needed.

Example: The width of a hex head bolt is 0.3123 inches. Convert the decimal 0.3123 to a common fraction to decide which socket would be the best fit for the bolt head. First, multiply the 0.3123 decimal by 64:

Next, round the product to the nearest whole number: 19.98722 ≈ 20.

Use this whole number (20) as the numerator and 64 as the denominator: 20⁄64.

Now, reduce 20⁄64 to 5⁄16.

Therefore, the correct socket would be the 5⁄16 inch socket (20⁄64 reduced).

Example: When accurate holes of uniform diameter are required for aircraft structures, they are first drilled approximately 1⁄64 inch undersized and then reamed to the final desired diameter. What size drill bit should be selected for the undersized hole if the final hole is reamed to a diameter of 0.763 inches? First, multiply the 0.763 decimal by 64.

Next, round the product to the nearest whole number: 48.832 ≈ 49.

Use this number (49) as the numerator and 64 as the denominator: 49⁄64 is the closest fraction to the final reaming diameter of 0.763 inches. To determine the drill size for the initial undersized hole, subtract 1⁄64 inch from the finished hole size.

Therefore, a 3⁄4-inch drill bit should be used for the initial undersized holes.

Converting Fractions to Decimals 

To convert any fraction to a decimal, simply divide the top number (numerator) by the bottom number (denominator). Every fraction will have an approximate decimal equivalent.


Calculator tip: Numerator (top number) ÷ Denominator (bottom number) = the decimal equivalent of the fraction.

Some fractions when converted to decimals produce a repeating decimal.


Decimal Equivalent Chart

Figure 1-6 is a fraction to decimal to millimeter equivalency chart. Measurements starting at 1⁄64 inch and up to 23 inches have been converted to decimal numbers and to millimeters.

Figure 1-6. Fractions, decimals, and millimeters.

Figure 1-6. Fractions, decimals, and millimeters.