Introduce operand offset (C++ and Java)

[kkaempf]

This PR introduces a new pre-defined symbol named operand_offset to expose the relative byte offset of the current operand from inst_start. This is required to correctly compute PC-relative operand addresses for DEC VAX.

See #4606 about my attempts of solving this (and failing) with existing methods.

While the diff looks relatively large, there's no new computation going on. It merely exposes the offset which parserWalker already has. The rest of the code is just "symbol ceremony" copied from inst_start.

There are two commits, the first one covers the "C++" side, the second covers the "Java" side.

Fixes #4606

[jobermayr]

Did you forget to
git add Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/context/ParserWalker.java ?

[kkaempf]

Did you forget to git add Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/context/ParserWalker.java ?

🤔 - I don't have this file (and I don't need it either).

The two commits are the two patches I use to build Ghidra 10.1.5. (Update to 10.2.2 is in progress 😉).

[jobermayr]

Thanks for the hint. 39baf3a removed ParserWalker.
Current squashed patch to build against "master":
https://github.com/jobermayr/ghidra-staging/blob/master/4812-Introduce-operand-offset-C-and-Java.patch

[kkaempf]

Rebased to head

[jobermayr]

You did not take my comment from 12/12/2022 into account:

> Task :Decompiler:compileSleighLinux_x86_64ExecutableSleighCpp
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/slghscan.cc: In function ???ghidra::int4 ghidra::find_symbol()???:
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/slghscan.cc:1578:5: error: ???yylval??? was not declared in this scope
 1578 |     yylval.offsetsym = (OffsetSymbol *)sym;
      |     ^~~~~~
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/slghscan.cc: At global scope:
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/slghscan.cc:2965:17: warning: ???void yyunput(int, char*)??? defined but not used [-Wunused-function]
 2965 |     static void yyunput (int c, char * yy_bp )
      |                 ^~~~~~~

/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.cc:2929:7: error: no declaration matches ???ghidra::uintb ghidra::PcodeSnippet::allocateTemp()???
 2929 | uintb PcodeSnippet::allocateTemp(void)
      |       ^~~~~~~~~~~~
In file included from /tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.cc:88:
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.hh:80:17: note: candidate is: ???virtual ghidra::uint4 ghidra::PcodeSnippet::allocateTemp()???
   80 |   virtual uint4 allocateTemp(void);
      |                 ^~~~~~~~~~~~
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.hh:72:7: note: ???class ghidra::PcodeSnippet??? defined here
   72 | class PcodeSnippet : public PcodeCompile {
      |       ^~~~~~~~~~~~


> Task :SoftwareModeling:compileJava FAILED
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghsymbol/OffsetSymbol.java:20: error: cannot find symbol
import ghidra.pcodeCPort.context.ParserWalker;
                                ^
  symbol:   class ParserWalker
  location: package ghidra.pcodeCPort.context
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghsymbol/OffsetSymbol.java:76: error: cannot find symbol
        public void getFixedHandle(FixedHandle hand, ParserWalker pos) {
                                                     ^
  symbol:   class ParserWalker
  location: class OffsetSymbol
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghsymbol/OffsetSymbol.java:84: error: cannot find symbol
        public void print(PrintStream s, ParserWalker pos) {
                                         ^
  symbol:   class ParserWalker
  location: class OffsetSymbol
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghpatexpress/OffsetInstructionValue.java:20: error: cannot find symbol
import ghidra.pcodeCPort.context.ParserWalker;
                                ^
  symbol:   class ParserWalker
  location: package ghidra.pcodeCPort.context
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghpatexpress/OffsetInstructionValue.java:35: error: cannot find symbol
        public long getValue(ParserWalker pos) {
                             ^
  symbol:   class ParserWalker
  location: class OffsetInstructionValue
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghsymbol/OffsetSymbol.java:75: error: method does not override or implement a method from a supertype
        @Override
        ^
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghsymbol/OffsetSymbol.java:83: error: method does not override or implement a method from a supertype
        @Override
        ^
/tmp/ghidra/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/pcodeCPort/slghpatexpress/OffsetInstructionValue.java:34: error: method does not override or implement a method from a supertype
        @Override
        ^
8 errors

> Task :Decompiler:compileSleighLinux_x86_64ExecutableSleighCpp FAILED

[kkaempf]

Sorry, @jobermayr, indeed ParserWalker moved and I didn't properly test after rebasing.

Fixed now.

[jobermayr]

> Task :Decompiler:compileSleighLinux_x86_64ExecutableSleighCpp
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.cc:2929:7: error: no declaration matches ???ghidra::uintb ghidra::PcodeSnippet::allocateTemp()???
 2929 | uintb PcodeSnippet::allocateTemp(void)
      |       ^~~~~~~~~~~~
In file included from /tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.cc:88:
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.hh:80:17: note: candidate is: ???virtual ghidra::uint4 ghidra::PcodeSnippet::allocateTemp()???
   80 |   virtual uint4 allocateTemp(void);
      |                 ^~~~~~~~~~~~
/tmp/ghidra/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.hh:72:7: note: ???class ghidra::PcodeSnippet??? defined here
   72 | class PcodeSnippet : public PcodeCompile {
      |       ^~~~~~~~~~~~


> Task :Decompiler:compileSleighLinux_x86_64ExecutableSleighCpp FAILED

Fix (edit: previously it was to generated pcodeparse.cc):

diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.y b/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.y
index 1eee53d8f..b7d8288c8 100644
--- a/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.y
+++ b/Ghidra/Features/Decompiler/src/decompile/cpp/pcodeparse.y
@@ -650,7 +650,7 @@ void PcodeLexer::initialize(istream *t)
   }
 }
 
-uintb PcodeSnippet::allocateTemp(void)
+uint4 PcodeSnippet::allocateTemp(void)
 
 { // Allocate a variable in the unique space and return the offset
   uint4 res = tempbase;

[kkaempf]

Thanks @jobermayr - I had fixed this in the generated file only 🤦🏻

Can you point me to your build system ? It seems as if you've extended "make realclean" within Ghidra/Features/Decompiler/src/decompile/cpp 🤔

[jobermayr]

Can you point me to your build system ? It seems as if you've extended "make realclean" within Ghidra/Features/Decompiler/src/decompile/cpp

Not really. I only created a makeBison.sh to quickly regenerate files after modifications to .l or .y files.
I never do a make realclean or so. I only "clone" and apply patches from ghidra-staging after restarting my PC and RAM (64 GB :) :) ) gets cleared then.

On ghidra-staging I try to keep all pull requests (and their updates) building.
Sometimes I also find runtime errors, try to fix them and report back :)

[jobermayr]

To apply and build after 8fbd171 / ae6f7b4: https://github.com/jobermayr/ghidra-staging/blob/master/4812-Introduce-operand-offset-C-and-Java.patch
@kkaempf Can you please check it?

[kkaempf]

To apply and build after 8fbd171 / ae6f7b4:

I'd assume these commits are included in Ghidra_11.0.2_build ?! 🤔

https://github.com/jobermayr/ghidra-staging/blob/master/4812-Introduce-operand-offset-C-and-Java.patch @kkaempf Can you please check it?

It looks wrong to me. For example:

Your patch:

 .../Decompiler/src/decompile/cpp/pcodeparse.y |   1 +

mine (also here):

 .../Decompiler/src/decompile/cpp/pcodeparse.y |  6 ++

I carry my commits along at my own fork, currently at (resp. on top of) Ghidra_11.0.2_build.

See https://build.opensuse.org/package/show/security:forensics/ghidra for the patches.

[jobermayr]

To apply and build after 8fbd171 / ae6f7b4:

I'd assume these commits are included in Ghidra_11.0.2_build ?! 🤔

No. master which leads currently to 11.1

https://github.com/jobermayr/ghidra-staging/blob/master/4812-Introduce-operand-offset-C-and-Java.patch @kkaempf Can you please check it?

It looks wrong to me. For example:

Your patch:

 .../Decompiler/src/decompile/cpp/pcodeparse.y |   1 +

mine (also here):

 .../Decompiler/src/decompile/cpp/pcodeparse.y |  6 ++

I try to collapse it like ae6f7b4:

Doesn't it make sense?

[kkaempf]

I will need to take a closer look. Not sure I can manage it this week. Asking for your patience 😉

[kkaempf]

I usually rebase my branch only on major releases.
I'll need to do a test-rebase to main and check the results.