Pass values from C program to Assembly language

I would like to pass values from C program to Assembly using the linked assembly method instead of inline assembly method in C. Below is the Assembly program(GCD) which is am working on.

;gcdasm.nasm
bits 64
section .text
global gcdasm
gcdasm:
    push rbp
    mov rbp, rsp
    mov rax, [rbp+4]        ;load rax with x
    mov rbx, [rbp+8]        ;load rbx with y
top:
    cmp rax, rbx            ;x(rax) has to be larger than y(rbx)
    je exit                 ;if x=y then exit and return value y
    jb xchange              ;if x<y then swap x and y
modulo:
    cqo                     ;RDX:RAX sign extend
    div rbx                 ;div rdx:rax with rbx
    cmp rdx, 0              ;check remider if its 0
    je exit                 ;if reminder is 0 then exit return return y
    mov rax, rdx            ;reminder rdx as next dividend
    jmp modulo              ;loop 
xchange:
    xchg rax, rbx           ;swap x and y
    jmp modulo

exit:
    mov rax, rbx            ;Return c program with the divisor y
    mov rsp, rbp
    pop rbp
    ret

And this is the C program from with I am trying to pass the values to assembly program

//gcd.c
#include<stdio.h>

extern int gcdasm(int x, int y); 

int main(void){
    int x=0;
    int y=0;
    int result=0;

    x = 46; 
    y = 90; 
    printf("%d and %d have a gcd of %dn", x,y,gcdasm(x,y));

    x = 55;
    y = 66;
    printf("%d and %d have a gcd of %dn", x,y,gcdasm(x,y));

    return 0;
}

When I compile using the below method and run it. I get either error Floating point exception or an empty prompt waiting for input

$ nasm -felf64 gcdasm.nasm -o gcdasm.o
$ gcc gcdasm.o gcd.c -o gcd
$ ./gcd 
Floating point exception
$ ./gcd 

I am unable to figure out the error. Kindly help me out. Thank you.

Answer

Passing arguments to gcdasm()

The two int arguments are passed through registers, not the stack. The first and second arguments are passed in the lower-half of rdi and rsi (i.e.: edi and esi), respectively. So, by sign extending edi and esi into rax and rbx respectively, you load the passed arguments into those registers:

movsx rax, edi  ;load rax with x
movsx rbx, esi  ;load rbx with y

However, note that rbx is not a scratch register, therefore the callee needs to save it before modifying it and then restore it back before leaving the gcdasm function.

You can simply replace rbx by rcx (which isn’t a callee-saved register) everywhere in your code. You don’t need rbp at all, so you can remove all the instructions where rbp appears.

---

Other problems

• There is also a problem with the logic of the program with:

  mov rax, rdx   ;reminder rdx as next dividend
  

  Instead of this, the divisor (rcx) should become the dividend (rax) and the remainder (rdx) should become the divisor (rcx), that is:

  mov rax, rcx
  mov rcx, rdx
  

• When dividing signed values, you have to use the idiv instruction, not div.

---

Improvement

There are also some reasons regarding performance and code size to use test rdx, rdx instead of cmp rdx, 0 for comparing rdx against zero.

---

With all that above in mind:

;gcdasm.nasm
bits 64
section .text
global gcdasm
gcdasm:
    movsx rax, edi          ;load rax with x
    movsx rcx, esi          ;load rcx with y
top:
    cmp rax, rcx            ;x(rax) has to be larger than y(rcx)
    je exit                 ;if x=y then exit and return value y
    jb xchange              ;if x<y then swap x and y
modulo:
    cqo                     ;sign extend RDX:RAX
    idiv rcx                ;rdx:rax/rcx (signed values)
    test rdx, rdx           ;check whether remainder is zero
    je exit                 ;if reminder is 0 then exit return y
    mov rax, rcx            ;divisor becomes dividend
    mov rcx, rdx            ;remainder becomes divisor
    jmp modulo              ;loop 
xchange:
    xchg rax, rcx           ;swap x and y
    jmp modulo

exit:
    mov rax, rcx            ;Return c program with the divisor y
    ret